11 Commits

Author SHA1 Message Date
Mayur Singal
7760663b22
MINOR: Change ingestion licence header (#20549) 2025-04-03 10:39:47 +05:30
Imri Paran
cd74d8f55a
MINOR: ref(data-quality): modularized test case validator import (#18716)
* ref(data-quality): modularized test case validator import

- removed test_suite_factory
- implemented TestCaseImporter
- removed SQAValidatorBuilder and PandasValidatorBuilder in favor of a SourceType enum
- removed the orm table creation from test suite source

* format

* IValidatorBuilder -> ValidatorBuilder

* use the table from the sampler in the test suite interface

* linting

* fixed the profiler with similar solution

* removed unused inheritance

* removed unneeded super().__init__()

* removed all instances of orm_table

* fixed tests

* add reportExplicitAny=false

* fixed tests
2024-11-27 16:25:12 +01:00
Teddy
58699063db
MINOR -- Fix DQ Partition Issue (#18641)
* fix: renamed `random_sample` to `get_dataset` and change dunder method access for SQA Table object

* fix: removed handle_partition decorator

* fix: fixed DQ partition issue + moved to `tablesample` method

* style: ran python linting

* style: fix python format check issues

* feat: added postgres tablesample

* style: ran python linting

* fix: sampling delta

* fix: merge conflicts

* fix: resolved conflicts

* style: ran python linting

* fix: patch orm call in test case

* fix: mock build_table_orm call in tests

* fix: test case failures and errors

* fix: removed unused import

* fix: patch typo

* fix: trino table schema retrieval

* fix: remove tuple context manager for 3.8 test support
2024-11-27 08:50:54 +01:00
Pere Miquel Brull
c68a45e7d8
Create new Auto Classification Workflow (#18610) 2024-11-19 08:10:45 +01:00
Teddy
d579008c99
GEN 1683 - Add Column Value to be At Expected Location Test (#18524)
* feat: added column value to be in expected location test

* fix: renamed value -> values

* doc: added 1.6 documentatio entry

* style: ran python linting

* fix: move data packaging to pyproject.yaml

* fix: add init file back for data package

* fix: failing test case
2024-11-06 11:17:13 +01:00
Ayush Shah
a15da7ec98
Issue #14812: Add support for empty string as missing count (#16017) 2024-04-25 09:45:26 +05:30
Teddy
3dc642989c
Fixes #7729 - Add logic to compute passed/failed rows (#14472)
* feat: add test case resolution task workflow

* chore: add migration for test case resolution feature

* fix: removed required field for object compatibiity in older migrations

* fix: minor testCaseResolution status logic

* chore: revert migration for test case incident

* chore: update migration file

* style: renamed variables

* feat: added logic to compute failed/passed rows

* feat: add support for row level computation in schema

* chore: add test definition migration

* feat: add logic to explicitly compute row level failure

* chore: clean up code

* style: fix java

* style: fix pyton format

* fix: unhidde API for incident manager

* style: fix java styling
2023-12-27 13:38:51 +01:00
Teddy
d0cffdcd66
Fixes Issue #11438 - Implement threshold and startegy for custom SQL (#11847)
* feat: Add threshold and strategy logic on the custom SQL object test

* feat: ran python linting

* feat: added safety checks for custom sql query

* feat: ran python linting
2023-06-02 09:41:31 +02:00
Teddy
9b4e9132ae
fixed #9656 - Add support for date type to column values to be between (#10890)
* fix: renamed  to  submodule

* fix: linting

* fix: columnValuesToBeBetween test for date column type
2023-04-04 17:16:44 +02:00
Teddy
83be5d933b
Fixes #9301 - Refactor TestSuite and Remove Pandas from Base Requirements (#10244)
* feat(testSuite): extracted out column test for SQA type

* refactor(testSuite): extracted SQA column and table tests into their own classes

* refactor(testSuite): Added pkutil namespace package style for test suite classes

* refactor(testSuite): added dynamic importer function for test cases

* refactor(testSuite): black formatting

* refactor(testSuite): fixed linting issues

* refactor(testSuite): refactor metrics for dataframe

* refactor(testSuite): Added Mixins and base methods

* refactor(testSuite): extrcated out get bound for floats

* refactor(testSuite): Added pandas column test cases

* refactor(testSuite): Deleted old column tests

* refactor(testSuite): Added table tests for datalake

* refactor(testSuite): Removed old tests definition

* refactor(testSuite): changed registry to dynamic class inport

* refactor(testSuite): renamed dl_fn to df_fn

* refactor(testSuite): updated registry unit test

* refactor(testSuite): updated import path to sqa like column

* refactor(testSuite): cleaned up imports in old files

* refactor(testSuite): harmonzied SQALikeColumn object to replicate SQA Column object

* refactor(testSuite): linting

* refactor(testSuite): linting

* refactor(testSuite): raise expection on DQ exception

* refactor(testSuite): linting

* refactor(testSuite): removed pandas from base requirements

* refactor(testSuite): Added __futur__ for py3.7 type hint

* refactor(testSuite): added `df` to good-names

* refactor(testSuite): renamed Handler to Validator

* refactor(testSuite): Added test inheritance for column tests

* refactor(testSuite): cleaned up column type check

* refactor(testSuite): cleaned up typo

* refactor(testSuite): extracted main table test logic into parent class

* refactor(testSuite): linting

* refactor(testSuite): linting fixes

* refactor(testSuite): address doc string and linting issues
2023-02-22 09:42:34 +01:00
Teddy
ba08302ea1
Issue #7291 - Implements Table Rows Inserted to be Between test (#9813)
* staging commit

* staging commit

* refactor: partitioning logic

* refactor (tests): move to parametrized tests for test validations

* refactor: local variables into global

* (feat): Added logic for table row inserted test

* (feat): fix python checkstyle

* feature: extracted get_query_filter logic into its own function
2023-01-31 15:57:51 +01:00