13 Commits

Author SHA1 Message Date
Pere Miquel Brull
6a3ce624b7
Fix #9008 - Add Airflow callback, introduce Workflow Status Mixin and remove retries (#9049)
Fix #9008 - Add Airflow callback, introduce Workflow Status Mixin and remove retries (#9049)
2022-11-30 08:30:45 +01:00
NiharDoshi99
d783869ed6
Fix: Added metadata service (#8759)
* Fix: Added metadata service

* Fix as per comment

* Fix changes for py-test

* Fix changes for py-test

* Fix py-checkstyle
2022-11-16 06:41:54 +01:00
Teddy
3dbaa69978
Data insight workflow (#8729) 2022-11-15 05:44:25 +01:00
Nahuel
91d0460b27
Fix#8577: Refactor part of the secrets manager implementation (#8617)
* Removed part of the secrets manager implementation in the server side

* Removed part of the secrets manager implementation in the openmetadata python library side

* Remove deprecated test

* Address pylint checks

* Address new pylint checks

* Address PR comments

* Fix import on airflows apis

* Clear singleton instances for running Ometa secrets manager test
2022-11-11 09:59:15 +01:00
Mayur Singal
1d93d0e650
Fix #8570: Parsing Workflow Gracefully on Deploy (#8585)
* Fix #8570: Parsing Workflow Gracefully on Deploy

* use entity_class

* py format
2022-11-09 08:30:22 +01:00
Mayur Singal
1386b43607
Fix #6141: Ingestion Pipeline Status Updates (#8216)
* Fix #6141: Ingestion Pipeline Status Updates

* List Pipeline Status API & Improvements

* Rename State field to PipelineState in UI

* Convert Pipeline Status array to single object

* fix braking UI

* Rebase Fixes

* Profiler, TestStuite & DataInsigts Pipeline

* py_format

* fix logs page not loading
add pipelineStatus endpoint

* fix recent run changes

* Fix Tests

* address review comments for ui

* fix failing checks

* fix unit tests

* fix cypress test

* remove loader test as not using it

* wait for API when we click on the tab

* fix cypress waitFor api

* fix failed cypress tests

Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
2022-11-03 14:37:26 +05:30
Teddy
83b048c07c
Fixes #8206 -- Implement data insight refinement for entity report data (#8351)
* added data insight module to refine and ingest data

* Added json schema for data insight workflow

* Added mixin for data insight

* Added tests for data insight workflow and methods

* Fixed python style

* Fixed python linting

* Fixed unit test

* Added data insight worflow + workflow logging

* Fix python style

* Renamed analytic -> data_insight + added test to make + fixed style

* Fixed java style

* Fixed code smells + python style

* Ignore ES linting for now

* Fixed comments from review

* fix python formatting

* Fixe PipelineType attribute for data insight
2022-10-26 11:18:08 +02:00
Teddy
595e5c1b89
Added checks to see if DAG can reach OM (#7794) 2022-09-30 17:57:03 +02:00
Shailesh Parmar
3b67cc824d
Data qaulity add test's workflow (#6957)
* initial setup for data quality form

* added stepper and form component

* added select/add test suite step form

* added form for table test and cron step

* added data quality table test form flow

* bug fix for profiler

* added column test form

* render right panel information dynamically

* updated test as per new changes

* updated data test id

* Fixed ingestionPipeline

* Fixed pytest + python format

* miner fix, added sql editor

* removed filter for duplicate test check

* miner fix

Co-authored-by: Teddy Crepineau <teddy.crepineau@gmail.com>
2022-08-29 17:26:59 +05:30
Nahuel
bdbbca0efe
Fix#6027: Improve logging in OpenMetadata Airflow APIs (#6920)
Fix#6027: Improve logging in OpenMetadata Airflow APIs (#6920)
2022-08-26 07:29:38 +02:00
Teddy
ce578e73d4
Fixes #5831 by implenting testSuite workflow logic (#6911)
* Added database filter in workflow

* Removed association between profiler and data quality

* fixed tests with removed association

* Fixed sonar code smells and bugs

* Updated profiler workflow to:
- support only running profiler (removed test run)
- support column inclusion and exclusion
- added back support for partitioned table and sample

* moved status to workflow

* Fixed tests

* removed test logic from profiler sink

* Added logic to return sample from workflow sample value

* Added profiler examples

* Updated documentation for profiler

* Fixed code smells

* commited changed to profiler

* initial commit of the revamp workflow

* Fixed python formating

* cleaned up profiler submodule by removing test related files and functions

* Added airflow DAG logic for testSuite workflow

* Fixed code smells + added airflow ingestion tests + fixed comments
2022-08-25 10:01:28 +02:00
Nahuel
a878aa911c
Fix#6212: Retrieve connection params from secret manager in CLI commands (#6441)
* Retrieve connection params from secret manager for database connectors

* Retrieve connection params from secret manager for all services except database connectors

* Stop retrieving connection from SM in Airflow rest plugin

* Retrieve connection params from secret manager for dashboard services

* Retrieve connection params when initializing Workflow/ProfilerWorkflow objects

* Align services topologies + comment changes in topology runner

* Address SonarCloud bug detected

* Update database service topology

* Address PR comments

* Address PR comments

* Address PR comments
2022-08-02 09:13:46 +02:00
Pere Miquel Brull
b7eb41a560
Fix #5840 - Upgrade Airflow container to 2.3.3 (#6362)
Fix #5840 - Upgrade Airflow container to 2.3.3 (#6362)
2022-07-28 14:46:25 +02:00