Mayur Singal
7760663b22
MINOR: Change ingestion licence header ( #20549 )
2025-04-03 10:39:47 +05:30
Pere Miquel Brull
c68a45e7d8
Create new Auto Classification Workflow ( #18610 )
2024-11-19 08:10:45 +01:00
Pere Miquel Brull
d8e2187980
#15243 - Pydantic V2 & Airflow 2.9 ( #16480 )
...
* pydantic v2
* pydanticv2
* fix parser
* fix annotated
* fix model dumping
* mysql ingestion
* clean root models
* clean root models
* bump airflow
* bump airflow
* bump airflow
* optionals
* optionals
* optionals
* jdk
* airflow migrate
* fab provider
* fab provider
* fab provider
* some more fixes
* fixing tests and imports
* model_dump and model_validate
* model_dump and model_validate
* model_dump and model_validate
* union
* pylint
* pylint
* integration tests
* fix CostAnalysisReportData
* integration tests
* tests
* missing defaults
* missing defaults
2024-06-05 21:18:37 +02:00
Teddy
056e6368d0
Issue #14765 - Preparatory Work ( #15312 )
...
* refactor!: change partition metadata structure for table entities
* refactor!: updated json schema for TypeScript code gen
* chore: migration of partition for table entities
* style: python & java linting
* updated ui side change for table partitioned key
* miner fix
* addressing comments
* fixed ci error
---------
Co-authored-by: Shailesh Parmar <shailesh.parmar.webdev@gmail.com>
2024-02-28 07:11:00 +01:00
Teddy
61ef55290e
MINOR - generic profiler optimization for sampling and BQ ( #14507 )
...
* fix: limit sampling to specific column
* fix: handle bigquery struct columns
* fix: default partition to 1 DAY for BQ
* fix: default to __TABLES__ for BQ table metrics
* style: ran python linting
* style: fix linting
* fix: python style
* fix: set partition to DAY if not HOUR
2023-12-27 19:13:44 +01:00
Pere Miquel Brull
a3bfd4e696
Part of #11968 - Restructure Profiler Workflow and PII Processor ( #13059 )
...
* Structure PII
* Restructure Profiler Workflow
* Update signature for abc
* remove profiler sink
* Fix tests
* Fix lint
* Fix test
* Fix test
2023-09-04 11:02:57 +02:00
Teddy
b89cf64f14
Clean up profiler ( #12369 )
...
* ref: implemented interface for profiler components + removed struct logic
* ref: ran python linting
* ref: added UML diagram to readme.md
* ref: empty commit for labeler check
* ref: remove multiple context manager for 3.7 3.8 compatibility
* ref: remove
2023-07-12 17:02:32 +02:00
Ayush Shah
65f370e4aa
Rename GCS to GCP ( #11812 )
2023-06-06 11:57:00 +05:30
Teddy
9b4e9132ae
fixed #9656 - Add support for date type to column values to be between ( #10890 )
...
* fix: renamed to submodule
* fix: linting
* fix: columnValuesToBeBetween test for date column type
2023-04-04 17:16:44 +02:00
Teddy
754074f1be
Fixes #7758 - Added Column value and Integer Range Partitionning ( #10350 )
...
* feat(profiler): renamed module to
* feat(profiler): added dbt-artifacts-parser to test setup.py
* feat(profiler): refactor workflow and interface
* feat(profiler): linting
* feat(profiler): removed old profiler modules
* feat(profiler): added support for value and integer range partition
* feat(profiler): fixed linting
* feat(profiler): added partitionning support for datalake profiler
* feat(profiler): removed `ProfilerInterfaceArgs` class
* feat(profiler): address comments
* feat(profiler): Added `OTHER` as an `IntervalType` for UI type generation
2023-03-01 08:20:38 +01:00
Milan Bariya
7f111584b8
Deprecate OMetaDatabaseAndTable ( #9760 )
...
* Deprecate OMetaDatabaseAndTable
* Change Based On Comments
* Change Based On Comments
* Fix Pytest
* Fix Pytest
2023-01-18 14:29:32 +01:00
Pere Miquel Brull
7f21a7bced
Fix #8088 - Restructure source connections & clients ( #9545 )
2023-01-02 13:52:27 +01:00
Ayush Shah
231b28fc87
Fix 7278: add test connection ( #9196 )
...
* Add Test Connection to Profiler
* remove Code Lint
* Fix AttributeError
* Fix Pytest
* Fix Bigquery Partition tests
* Fix Lint
2022-12-10 17:16:28 +01:00
Teddy
199b342288
Fixes #8135 - Implement partitioning config for profiler ( #8623 )
...
* Added logic to handle partitioning config in profiler
* extracted get_partition_details out of workflow classes
2022-11-10 10:54:31 +01:00
Teddy
3b7f576d04
Fixed linting for Profiler ( #7922 )
...
* - Fixed linting
- Added logic to skip partition check when engine is not BQ
- Added ingestion partition logic to testSuite
* Fixed python formating
* Fixed test for BQ partition
2022-10-11 09:36:36 +02:00
Nahuel
df909703eb
Fix#6615: Fix failing test still in CI ( #7725 )
...
* Fix failing test still some tests are failing
* Fix bug in service entity repository
* Fix all tests except Ingestion Pipeline integration tests
* Fix Ingestion Pipeline integration tests
* Fix failing Python tests
2022-09-26 20:39:42 +02:00
Ayush Shah
56c71c4101
Fix Ingestion Time Partitioned Tables on Profiler ( #7506 )
2022-09-26 19:41:40 +05:30
