Pere Miquel Brull
3186937cc2
MINOR - Update Auto Classification defaults for sample data & classif… ( #20587 )
...
* MINOR - Update Auto Classification defaults for sample data & classification
* fix tests
2025-04-07 15:56:57 +02:00
Mayur Singal
7760663b22
MINOR: Change ingestion licence header ( #20549 )
2025-04-03 10:39:47 +05:30
Ayush Shah
7a3990f350
Fixes 19119: Enhance TableCustomSQLQueryValidator to support threshold operation ( #20307 )
2025-03-27 13:11:56 +05:30
Pere Miquel Brull
e56f477a4a
Fix #19147 - Executable Test Suites ( #19221 )
...
* backend
* format & tests
* rename backend
* migrations and ingestion
* format & tests
* format & tests
* tests
* format & tests
* tests
* updated ui side of changes
* addressing comment
* fixed failing unit test
* fix test list
* added e2e test, and fixed existing test
---------
Co-authored-by: Shailesh Parmar <shailesh.parmar.webdev@gmail.com>
2025-01-07 17:59:54 +01:00
Pere Miquel Brull
7aacfe032c
MINOR - FQN encoding in ometa_api, TestSuite pipeline creation & serialization of test case results ( #18877 )
...
* DOCS - Update ES config
* MINOR - Add missing FQN encoding & force types
* MINOR - Add missing FQN encoding & force types
* format
* fix tests
2024-12-02 17:17:21 +01:00
Teddy
58699063db
MINOR -- Fix DQ Partition Issue ( #18641 )
...
* fix: renamed `random_sample` to `get_dataset` and change dunder method access for SQA Table object
* fix: removed handle_partition decorator
* fix: fixed DQ partition issue + moved to `tablesample` method
* style: ran python linting
* style: fix python format check issues
* feat: added postgres tablesample
* style: ran python linting
* fix: sampling delta
* fix: merge conflicts
* fix: resolved conflicts
* style: ran python linting
* fix: patch orm call in test case
* fix: mock build_table_orm call in tests
* fix: test case failures and errors
* fix: removed unused import
* fix: patch typo
* fix: trino table schema retrieval
* fix: remove tuple context manager for 3.8 test support
2024-11-27 08:50:54 +01:00
Pere Miquel Brull
c68a45e7d8
Create new Auto Classification Workflow ( #18610 )
2024-11-19 08:10:45 +01:00
Imri Paran
a3d6c1dd20
MINOR: tests(datalake): use minio ( #17805 )
...
* tests(datalake): use minio
1. use minio instead of moto for mimicking s3 behavior.
2. removed moto dependency as it is not compatible with aiobotocore (https://github.com/getmoto/moto/issues/7070#issuecomment-1828484982 )
* - moved test_datalake_profiler_e2e.py to datalake/test_profiler
- use minio instead of moto
* fixed tests
* fixed tests
* removed default name for minio container
2024-09-12 07:13:01 +02:00
Imri Paran
3069a63cb4
remove pandas import for null_ratio ( #17401 )
2024-08-12 17:20:11 +02:00
Matt Chamberlin
ac6ddbf6c4
MINOR: support JSONL datalake file types ( #16614 )
...
* fix: support JSONL datalake file types
* add jsonl zip file types
* update fileFormat enum in table schema
* add tests
* fix test data ref
* reformat
* fix tests
---------
Co-authored-by: Matthew Chamberlin <mchamberlin@ginkgobioworks.com>
Co-authored-by: Mayur Singal <39544459+ulixius9@users.noreply.github.com>
2024-06-21 09:54:19 +02:00
Pere Miquel Brull
d8e2187980
#15243 - Pydantic V2 & Airflow 2.9 ( #16480 )
...
* pydantic v2
* pydanticv2
* fix parser
* fix annotated
* fix model dumping
* mysql ingestion
* clean root models
* clean root models
* bump airflow
* bump airflow
* bump airflow
* optionals
* optionals
* optionals
* jdk
* airflow migrate
* fab provider
* fab provider
* fab provider
* some more fixes
* fixing tests and imports
* model_dump and model_validate
* model_dump and model_validate
* model_dump and model_validate
* union
* pylint
* pylint
* integration tests
* fix CostAnalysisReportData
* integration tests
* tests
* missing defaults
* missing defaults
2024-06-05 21:18:37 +02:00
Imri Paran
a4c516d2c7
Fixes 16305: Added Test Case for Matching Enum ( #16362 )
...
* Added Test Case for Matching Enum
1. Implemented the test case using the `matchEnum` parameter.
2. Added integration tests.
3. Added migrations.
* fix tests
* fixed tests
* format
* fixed tests
* clear search cache before running ingestion
* format
* changed scopt of aws fixture
* moved migrations to 1.5.0
2024-05-28 09:30:30 +02:00
Teddy
1cbdfb3ae7
Fixes #12601 - column filter for profiler workflow ( #13535 )
...
* fix: sample data ingestion to match entity profiler column setting
* fix: python linting
* fix: updated fn call
* fix: added logic to handle json filed in datalake connector
* fix: handle NA values in parsing
* fix: reverted sampler changes from #13338
* fix: reverted metric changes from #13338
* fix: added datalake profiler ingestion test
* fix: python linting
* fix: removed normalization of json blob in NoSQL db
2023-10-12 14:51:38 +02:00
