Pere Miquel Brull
b778bc7968
#14943 - Check tags before PII processor ( #15622 )
2024-03-21 14:15:28 +05:30
IceS2
7b20ed2f34
FIXES 15215: Add execution time tracker for ingestion ( #15013 )
...
* Initial idea on how to track execution times
* Fix linters
* Add missing decorators on the API Client
* Improve where the output is being handled
2024-02-16 09:58:20 +01:00
Teddy
9a4a9df836
Fix #14895 - Get Metadata from Parquet Schema ( #14956 )
...
* linting: fix python linting
* fix: get column types from parquet schema for parquet files
* style: python linting
* fix: remove displayType check in test as variation depending on OS
2024-02-01 09:02:52 +01:00
Pere Miquel Brull
b786064bc2
#11857 - Store workflow status in the Ingestion Pipeline Status ( #14462 )
...
* Register StackTraceError in spec
* Register StackTraceError in spec
* Register StackTraceError in spec
* Add todos
* Update status
* docs
* format
* Fix tests
* Fix tests
* Fix tests
* Ignore generated
* Fix tests
* Fix tests
* Tests
* Try constants
* Try constants
* Print
* Print
* Print
* order
* Fix service name
* fix ui error
---------
Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
2023-12-22 15:43:50 +01:00
Pere Miquel Brull
6fb3f12675
App Resource - Add /configure endpoint & WAII app definition ( #13983 )
...
* App Configuration
* TODO
* Configure App with OM server info
* Remove init
* Rename configs
* Remove extra source
2023-11-17 09:28:11 +01:00
Mayur Singal
b34111ea00
ElasticSearch: Fix ES connection ( #13919 )
2023-11-13 14:10:11 +05:30
Pere Miquel Brull
c742835766
Auto Tagger Application - Preparing the Ingestion Framework ( #13862 )
...
* Prepare the skeleton for generic app registration
* Prepare the skeleton for generic app registration
* Handle app runner
* Prepare the skeleton for generic app registration
* Prepare the skeleton for generic app registration
* Allow deployment
* Fix PII APP
* Fix lint
* Fix PII APP
* Fix PII APP
* Prepare config-based external apps
* Prepare config-based external apps
* Fix lint
* Prepare config-based external apps
* Fix DI errors
* Amend comments
2023-11-13 08:58:38 +01:00
Pere Miquel Brull
c6297b9cdf
Fix status handling for new services ( #13707 )
2023-10-25 20:48:14 +02:00
Pere Miquel Brull
899cd7e1fe
Fix DQ Workflow ( #13631 )
...
* Fix DQ Workflow
* Fix DQ Workflow
2023-10-18 11:49:38 +02:00
Pere Miquel Brull
d3da2d1b9f
Register Ingestion pipelines just from YAML ( #13501 )
...
* Register Ingestion pipelines just from YAML
* Format
2023-10-10 07:04:04 +02:00
Pere Miquel Brull
0282574bdd
Create ometa client once and pass it around & improve pycln config ( #13310 )
...
* Create ometa client once and pass it around & improve pycln config
* Fix
* Fix
* Fix tests
* Fix maven ci
* Fix tests
* Fix tests
* Fix tests
* Format
* Fix DI
2023-10-04 09:14:03 +02:00
Pere Miquel Brull
a3bfd4e696
Part of #11968 - Restructure Profiler Workflow and PII Processor ( #13059 )
...
* Structure PII
* Restructure Profiler Workflow
* Update signature for abc
* remove profiler sink
* Fix tests
* Fix lint
* Fix test
* Fix test
2023-09-04 11:02:57 +02:00
Pere Miquel Brull
6c0e9f5061
Part of #7272 - Centralize Workflows, Status, and Exception Management ( #13029 )
...
* Prep changes
* Prep changes
* prep changes
* Update imports
* Format
* Prep delete
* Prep delete
* Fix sink
* Prep test
* Commit
* passing either
* passing either
* Prep Either
* Metadata source with Either
* Update status
* Merge remote-tracking branch 'upstream/main' into issue-7272
* Format
* Linting
* Linting
* Linting
* Linting
* Fix tests
* Fix tests
* Fix tests
* Fix tests
* Fix tests
* Fix tests
* Fix tests
* Comments
2023-08-30 15:49:42 +02:00
