Pere Miquel Brull
7012e73d75
GEN-1166 - Improve Ingestion Workflow Error Summary ( #18280 )
...
* GEN-1166 - Improve Ingestion Workflow Error Summary
* fix test
* docs
* comments
2024-10-16 18:15:50 +02:00
Pere Miquel Brull
e820cb0450
GEN-1410 - Improve PII Logging information ( #17835 )
2024-09-16 07:05:30 +02:00
IceS2
c522f14178
MINOR: Refactor output_handlers to a WorkflowOutputHandler class ( #17149 )
...
* Refactor output_handlers to a WorkflowOutputHandler class
* Add old methods as deprecated to avoid breaking changes
* Extract WorkflowInitErrorHandler from workflow_output_handler
* Fix static checks
* Fix tests
* Fix tests
* Update code based on comments from PR
* Update comment
2024-07-29 09:20:34 +02:00
Pere Miquel Brull
2aef457785
FIX #16481 - Truncate ingestion pipeline status ( #16997 )
...
* FIX #16481 - Truncate ingestion pipeline status
* FIX #16481 - Truncate ingestion pipeline status
* FIX #16481 - Truncate ingestion pipeline status
2024-07-12 09:44:21 +02:00
Pere Miquel Brull
cb72a22b59
Fix - e2e tests for pydantic V2 ( #16551 )
...
* Fix - e2e tests for pydantic V2
* add correct default
* add correct default
* revert datetime aware
* revert datetime aware
* revert datetime aware
* revert datetime aware
* revert datetime aware
* revert datetime aware
* revert datetime aware
* revert datetime aware
* fix apis
* format
2024-06-06 19:36:17 -07:00
Pere Miquel Brull
d8e2187980
#15243 - Pydantic V2 & Airflow 2.9 ( #16480 )
...
* pydantic v2
* pydanticv2
* fix parser
* fix annotated
* fix model dumping
* mysql ingestion
* clean root models
* clean root models
* bump airflow
* bump airflow
* bump airflow
* optionals
* optionals
* optionals
* jdk
* airflow migrate
* fab provider
* fab provider
* fab provider
* some more fixes
* fixing tests and imports
* model_dump and model_validate
* model_dump and model_validate
* model_dump and model_validate
* union
* pylint
* pylint
* integration tests
* fix CostAnalysisReportData
* integration tests
* tests
* missing defaults
* missing defaults
2024-06-05 21:18:37 +02:00
Pere Miquel Brull
78324bd17c
MINOR - Set pipeline status early ( #16077 )
2024-04-30 09:22:04 +02:00
Pere Miquel Brull
b778bc7968
#14943 - Check tags before PII processor ( #15622 )
2024-03-21 14:15:28 +05:30
IceS2
7b20ed2f34
FIXES 15215: Add execution time tracker for ingestion ( #15013 )
...
* Initial idea on how to track execution times
* Fix linters
* Add missing decorators on the API Client
* Improve where the output is being handled
2024-02-16 09:58:20 +01:00
Teddy
9a4a9df836
Fix #14895 - Get Metadata from Parquet Schema ( #14956 )
...
* linting: fix python linting
* fix: get column types from parquet schema for parquet files
* style: python linting
* fix: remove displayType check in test as variation depending on OS
2024-02-01 09:02:52 +01:00
Pere Miquel Brull
b786064bc2
#11857 - Store workflow status in the Ingestion Pipeline Status ( #14462 )
...
* Register StackTraceError in spec
* Register StackTraceError in spec
* Register StackTraceError in spec
* Add todos
* Update status
* docs
* format
* Fix tests
* Fix tests
* Fix tests
* Ignore generated
* Fix tests
* Fix tests
* Tests
* Try constants
* Try constants
* Print
* Print
* Print
* order
* Fix service name
* fix ui error
---------
Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
2023-12-22 15:43:50 +01:00
Pere Miquel Brull
6fb3f12675
App Resource - Add /configure endpoint & WAII app definition ( #13983 )
...
* App Configuration
* TODO
* Configure App with OM server info
* Remove init
* Rename configs
* Remove extra source
2023-11-17 09:28:11 +01:00
Mayur Singal
b34111ea00
ElasticSearch: Fix ES connection ( #13919 )
2023-11-13 14:10:11 +05:30
Pere Miquel Brull
c742835766
Auto Tagger Application - Preparing the Ingestion Framework ( #13862 )
...
* Prepare the skeleton for generic app registration
* Prepare the skeleton for generic app registration
* Handle app runner
* Prepare the skeleton for generic app registration
* Prepare the skeleton for generic app registration
* Allow deployment
* Fix PII APP
* Fix lint
* Fix PII APP
* Fix PII APP
* Prepare config-based external apps
* Prepare config-based external apps
* Fix lint
* Prepare config-based external apps
* Fix DI errors
* Amend comments
2023-11-13 08:58:38 +01:00
Pere Miquel Brull
c6297b9cdf
Fix status handling for new services ( #13707 )
2023-10-25 20:48:14 +02:00
Pere Miquel Brull
899cd7e1fe
Fix DQ Workflow ( #13631 )
...
* Fix DQ Workflow
* Fix DQ Workflow
2023-10-18 11:49:38 +02:00
Pere Miquel Brull
d3da2d1b9f
Register Ingestion pipelines just from YAML ( #13501 )
...
* Register Ingestion pipelines just from YAML
* Format
2023-10-10 07:04:04 +02:00
Pere Miquel Brull
0282574bdd
Create ometa client once and pass it around & improve pycln config ( #13310 )
...
* Create ometa client once and pass it around & improve pycln config
* Fix
* Fix
* Fix tests
* Fix maven ci
* Fix tests
* Fix tests
* Fix tests
* Format
* Fix DI
2023-10-04 09:14:03 +02:00
Pere Miquel Brull
a3bfd4e696
Part of #11968 - Restructure Profiler Workflow and PII Processor ( #13059 )
...
* Structure PII
* Restructure Profiler Workflow
* Update signature for abc
* remove profiler sink
* Fix tests
* Fix lint
* Fix test
* Fix test
2023-09-04 11:02:57 +02:00
Pere Miquel Brull
6c0e9f5061
Part of #7272 - Centralize Workflows, Status, and Exception Management ( #13029 )
...
* Prep changes
* Prep changes
* prep changes
* Update imports
* Format
* Prep delete
* Prep delete
* Fix sink
* Prep test
* Commit
* passing either
* passing either
* Prep Either
* Metadata source with Either
* Update status
* Merge remote-tracking branch 'upstream/main' into issue-7272
* Format
* Linting
* Linting
* Linting
* Linting
* Fix tests
* Fix tests
* Fix tests
* Fix tests
* Fix tests
* Fix tests
* Fix tests
* Comments
2023-08-30 15:49:42 +02:00
