24 Commits

Author SHA1 Message Date
Imri Paran
d91273a30d
Fix 20325: Trigger external apps with config (#20397)
* wip

* feat: trigger external apps with override config

- Added in openmetadata-airflow-apis functionality to trigger DAG with feature.
- Modified openmetadata-airflow-apis application runner to accept override config from params.
- Added overloaded runPipeline with `Map<String,Object> config` to allow triggering apps with configuration. We might want to expand this to all ingestion pipelines. For now its just for apps.
- Implemented an example external app that can be used to test functionality of external apps. The app can be enabled by setting the `ENABLE_APP_HelloPipelines=true` environment variable.

* fix class doc for application

* fixed README for airflow apis

* fixes

* set HelloPipelines to disabeld by default

* fixed basedpywright errros

* fixed app schema

* reduced airflow client runPipeline to an overload with null config
removed duplicate call to runPipeline in AppResource

* Update openmetadata-docs/content/v1.7.x-SNAPSHOT/developers/applications/index.md

Co-authored-by: Matias Puerta <matias@getcollate.io>

* deleted documentation file

---------

Co-authored-by: Matias Puerta <matias@getcollate.io>
2025-05-06 17:41:24 +07:00
Mayur Singal
7760663b22
MINOR: Change ingestion licence header (#20549) 2025-04-03 10:39:47 +05:30
Pere Miquel Brull
8d739563f2
FIX CL-548 - Encrypt JWT Token w/ Secrets Manager (#16861)
* encrypt jwt

* encrypt jwt

* Handle token revokes with SM

* #10192 - Validate both include and excludes filters

* #10192 - Validate both include and excludes filters

* docs
2024-07-11 09:16:48 +02:00
IceS2
34f6f81a20
MINOR: added exclude_defaults=False for profiler and deploy on airflow-apis (#16734) 2024-06-21 12:05:13 +05:30
Pere Miquel Brull
cb72a22b59
Fix - e2e tests for pydantic V2 (#16551)
* Fix - e2e tests for pydantic V2

* add correct default

* add correct default

* revert datetime aware

* revert datetime aware

* revert datetime aware

* revert datetime aware

* revert datetime aware

* revert datetime aware

* revert datetime aware

* revert datetime aware

* fix apis

* format
2024-06-06 19:36:17 -07:00
Pere Miquel Brull
d8e2187980
#15243 - Pydantic V2 & Airflow 2.9 (#16480)
* pydantic v2

* pydanticv2

* fix parser

* fix annotated

* fix model dumping

* mysql ingestion

* clean root models

* clean root models

* bump airflow

* bump airflow

* bump airflow

* optionals

* optionals

* optionals

* jdk

* airflow migrate

* fab provider

* fab provider

* fab provider

* some more fixes

* fixing tests and imports

* model_dump and model_validate

* model_dump and model_validate

* model_dump and model_validate

* union

* pylint

* pylint

* integration tests

* fix CostAnalysisReportData

* integration tests

* tests

* missing defaults

* missing defaults
2024-06-05 21:18:37 +02:00
Pere Miquel Brull
ba5f929f77
Add fine-grained health validation for Airflow APIs (#11792)
* Add fine-grained health validation for Airflow APIs

* Add ingestion version response

* Improve messaging

* Format

* Format

* Update response

* ui:update the hook to return airflow status based on response status field

* ui: add unit test for useAirflowStatus hook

* chore: only show test connection if airflow is available

* feat: add airflow message banner

* chore: update icon and background color

* chore: update typography to text

* test: add unit test

* address comments

* chore: show banner on service detail page

* fix: update test suite api workflow to reflect new implementation

---------

Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com>
Co-authored-by: Teddy Crepineau <teddy.crepineau@gmail.com>
2023-06-12 07:47:45 +02:00
Pere Miquel Brull
50ad38ea0f
Fix #11548 - Secrets Managers comms with OMeta (#11602)
* Remove secretsManagerCredentials from backend

* Remove secretsManagerCredentials from backend

* Add secrets manager loader

* Load SM in the ometa client

* Fix tests

* Fix tests

* Fix Lint

* Mock AWS region

---------

Co-authored-by: Ayush Shah <ayush@getcollate.io>
2023-05-19 09:43:11 +02:00
Pere Miquel Brull
b480e853ef
Fix XSS warnings (#11620) 2023-05-18 14:51:06 +05:30
Ayush Shah
9d11029ec8
Fixes 10351: Fixes Metrics Computation, Samping, test suites and partioning (#10603)
Co-authored-by: Teddy Crepineau <teddy.crepineau@gmail.com>
2023-04-11 20:58:31 +05:30
Pere Miquel Brull
4853dfc6c8
Deprecate test_connection endpoint (#10798) 2023-03-28 15:52:36 +02:00
Pere Miquel Brull
477a5223eb
Fix #10401 - Add Automations Workflow Resource & PUT service test connection result (#10437)
Fix #10401 - Add Automations Workflow Resource & PUT service test connection result (#10437)
2023-03-06 14:44:16 +01:00
Milan Bariya
7faec7880a
Update test connection (#10317)
* Update test connection

* Update test connection

* Update test connection with Tags

* Change based on comments

* Fix Pytest

* Service connectin arg in all connecter

* Fix pytest and conflict
2023-03-02 14:22:38 +05:30
Milan Bariya
c1a8553e07
Airflow API change for test connection (#10182)
* Airflow API change for test connection

* Improve logic

* Pydantic change

* Improve logic
2023-02-24 11:43:11 +05:30
Pere Miquel Brull
567a62f7db
Fix session handling for refresh job (#9636)
Co-authored-by: NiharDoshi99 <51595473+NiharDoshi99@users.noreply.github.com>
2023-01-06 20:38:37 +05:30
Pere Miquel Brull
7f21a7bced
Fix #8088 - Restructure source connections & clients (#9545) 2023-01-02 13:52:27 +01:00
Nahuel
5151db4ae0
Fix bug in secrets manager with SSO configuration (#8932)
* Fix bug in secrets manager with SSO configuration

* Remove comment from imports

* Sort imports

* Add missing decrypt when updating

* Add minor change
2022-11-21 22:37:20 +01:00
Nahuel
91d0460b27
Fix#8577: Refactor part of the secrets manager implementation (#8617)
* Removed part of the secrets manager implementation in the server side

* Removed part of the secrets manager implementation in the openmetadata python library side

* Remove deprecated test

* Address pylint checks

* Address new pylint checks

* Address PR comments

* Fix import on airflows apis

* Clear singleton instances for running Ometa secrets manager test
2022-11-11 09:59:15 +01:00
Mayur Singal
a49c261cbe
Add Queued Status in Ingestion Pipelines (#8529) 2022-11-04 11:19:50 +05:30
Pere Miquel Brull
268d1171f1
Support Airflow 2.2.2 (#8273) 2022-10-21 19:42:52 +02:00
Pere Miquel Brull
339f123f92
Fix #7477 - Paginate Airflow logs (#8145)
* Paginate last dag logs

* clean logic

* Prep pagination

* Prep pagination

* Format

* Update docker/local-metadata/docker-compose.yml

Co-authored-by: Nahuel <nahuel@getcollate.io>

* UI : Add pagination support for airflow logs

* Addressing review comments

Co-authored-by: Nahuel <nahuel@getcollate.io>
Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com>
2022-10-21 17:32:56 +05:30
Nahuel
bdbbca0efe
Fix#6027: Improve logging in OpenMetadata Airflow APIs (#6920)
Fix#6027: Improve logging in OpenMetadata Airflow APIs (#6920)
2022-08-26 07:29:38 +02:00
Nahuel
4e176fbc66
Fix#5917: Implementation of temp secret for testing connection (#6832)
* Implementation of temp secret for testing connection

* Fix tests
2022-08-22 08:43:23 +02:00
Pere Miquel Brull
b7eb41a560
Fix #5840 - Upgrade Airflow container to 2.3.3 (#6362)
Fix #5840 - Upgrade Airflow container to 2.3.3 (#6362)
2022-07-28 14:46:25 +02:00