yujunjun/OpenMetadata
1
0
Fork 0
mirror of https://github.com/open-metadata/OpenMetadata.git synced 2025-10-14 02:08:54 +00:00
Code Issues Packages Projects Releases Wiki Activity
OpenMetadata/ingestion/src/metadata/data_quality/validations/models.py

27 lines
676 B
Python
Raw Normal View History

Fix #16421: add tableDiff test case (#16554) * feat: add tableDiff test case This changed introduces a "table diff" test case which compares two tables and fails if they are not identical. The similarity is made based on a specific "key" (because the test only makes sense when performed on ordered collections). 1. Added the `tableDiff` test definition. 2. Implemented a "runtime" parameters feature which injects additional parameters for the test at runtime. 3. Integration tests (because of course). This feature was not tested end-to-end yet because "array" data * pydantic v2 * format * format * format and added data diff to setup.py * format * fixed param issue which has type ARRAY * fixed runtime_parameter_setter * moved models to parent directory * handle errors in table diff * fixed issue with edit test case * format * added more details to pytest skip * format * refactor: Improve createTestCaseParameters function in DataQualityUtils * fixed unit test * removed unused fixture * removed validator.py * fixed tests * added validate kwarg to tests_mixin * removed "postgres" data diff extra as they interfere with psycopg2-binary * fixed tests * pinned tenacity for tests * reverted tenacity pinning * added ui support for test diff * fixed dq cypress and added edit flow * organized the test case * added dialect support * fixed tests * option style fix * fixed calculation for passing/failing rows * restrict the tableDiff test to limited services * set where to None if blank string * fixed where clause * fixed tests for where clause * use displayName in place of name in edit form * added docs for RuntimeParameterSetter * fixed cypress --------- Co-authored-by: Shailesh Parmar <shailesh.parmar.webdev@gmail.com>
2024-06-20 16:54:12 +02:00
"""Models for the TableDiff test case"""
MINOR: add column diff for table diff test case (#16809) * feat(table-diff): added column validation added column validation for table diff that will be carried out before running the row level diff. If a diff for the column exists, it will short circuit the test and report. * fixed unit tests * format * - resolve column types more robustly - changed test result metric to include "rows" or "columns"
2024-07-02 12:36:03 +02:00
Fix #16421: add tableDiff test case (#16554) * feat: add tableDiff test case This changed introduces a "table diff" test case which compares two tables and fails if they are not identical. The similarity is made based on a specific "key" (because the test only makes sense when performed on ordered collections). 1. Added the `tableDiff` test definition. 2. Implemented a "runtime" parameters feature which injects additional parameters for the test at runtime. 3. Integration tests (because of course). This feature was not tested end-to-end yet because "array" data * pydantic v2 * format * format * format and added data diff to setup.py * format * fixed param issue which has type ARRAY * fixed runtime_parameter_setter * moved models to parent directory * handle errors in table diff * fixed issue with edit test case * format * added more details to pytest skip * format * refactor: Improve createTestCaseParameters function in DataQualityUtils * fixed unit test * removed unused fixture * removed validator.py * fixed tests * added validate kwarg to tests_mixin * removed "postgres" data diff extra as they interfere with psycopg2-binary * fixed tests * pinned tenacity for tests * reverted tenacity pinning * added ui support for test diff * fixed dq cypress and added edit flow * organized the test case * added dialect support * fixed tests * option style fix * fixed calculation for passing/failing rows * restrict the tableDiff test to limited services * set where to None if blank string * fixed where clause * fixed tests for where clause * use displayName in place of name in edit form * added docs for RuntimeParameterSetter * fixed cypress --------- Co-authored-by: Shailesh Parmar <shailesh.parmar.webdev@gmail.com>
2024-06-20 16:54:12 +02:00
from typing import List, Optional
from pydantic import BaseModel
[GEN-1996] feat(data-quality): use sampling config in data diff (#18532) * feat(data-quality): use sampling config in data diff - get the table profiling config - use hashing to sample deterministically the same ids from each table - use dirty-equals to assert results of stochastic processes * - reverted missing md5 - added missing database service type * - use a custom substr sql function * fixed nounce * added failure for mssql with sampling because it requires a larger change in the data-diff library * fixed unit tests * updated range for sampling
2024-11-11 10:07:23 +01:00
from metadata.generated.schema.entity.data.table import Column, TableProfilerConfig
from metadata.generated.schema.entity.services.databaseService import (
DatabaseServiceType,
)
MINOR: add column diff for table diff test case (#16809) * feat(table-diff): added column validation added column validation for table diff that will be carried out before running the row level diff. If a diff for the column exists, it will short circuit the test and report. * fixed unit tests * format * - resolve column types more robustly - changed test result metric to include "rows" or "columns"
2024-07-02 12:36:03 +02:00
class TableParameter(BaseModel):
serviceUrl: str
path: str
columns: List[Column]
[GEN-1996] feat(data-quality): use sampling config in data diff (#18532) * feat(data-quality): use sampling config in data diff - get the table profiling config - use hashing to sample deterministically the same ids from each table - use dirty-equals to assert results of stochastic processes * - reverted missing md5 - added missing database service type * - use a custom substr sql function * fixed nounce * added failure for mssql with sampling because it requires a larger change in the data-diff library * fixed unit tests * updated range for sampling
2024-11-11 10:07:23 +01:00
database_service_type: DatabaseServiceType
MINOR: add column diff for table diff test case (#16809) * feat(table-diff): added column validation added column validation for table diff that will be carried out before running the row level diff. If a diff for the column exists, it will short circuit the test and report. * fixed unit tests * format * - resolve column types more robustly - changed test result metric to include "rows" or "columns"
2024-07-02 12:36:03 +02:00
Fix #16421: add tableDiff test case (#16554) * feat: add tableDiff test case This changed introduces a "table diff" test case which compares two tables and fails if they are not identical. The similarity is made based on a specific "key" (because the test only makes sense when performed on ordered collections). 1. Added the `tableDiff` test definition. 2. Implemented a "runtime" parameters feature which injects additional parameters for the test at runtime. 3. Integration tests (because of course). This feature was not tested end-to-end yet because "array" data * pydantic v2 * format * format * format and added data diff to setup.py * format * fixed param issue which has type ARRAY * fixed runtime_parameter_setter * moved models to parent directory * handle errors in table diff * fixed issue with edit test case * format * added more details to pytest skip * format * refactor: Improve createTestCaseParameters function in DataQualityUtils * fixed unit test * removed unused fixture * removed validator.py * fixed tests * added validate kwarg to tests_mixin * removed "postgres" data diff extra as they interfere with psycopg2-binary * fixed tests * pinned tenacity for tests * reverted tenacity pinning * added ui support for test diff * fixed dq cypress and added edit flow * organized the test case * added dialect support * fixed tests * option style fix * fixed calculation for passing/failing rows * restrict the tableDiff test to limited services * set where to None if blank string * fixed where clause * fixed tests for where clause * use displayName in place of name in edit form * added docs for RuntimeParameterSetter * fixed cypress --------- Co-authored-by: Shailesh Parmar <shailesh.parmar.webdev@gmail.com>
2024-06-20 16:54:12 +02:00
class TableDiffRuntimeParameters(BaseModel):
MINOR: add column diff for table diff test case (#16809) * feat(table-diff): added column validation added column validation for table diff that will be carried out before running the row level diff. If a diff for the column exists, it will short circuit the test and report. * fixed unit tests * format * - resolve column types more robustly - changed test result metric to include "rows" or "columns"
2024-07-02 12:36:03 +02:00
table1: TableParameter
table2: TableParameter
Fix #16421: add tableDiff test case (#16554) * feat: add tableDiff test case This changed introduces a "table diff" test case which compares two tables and fails if they are not identical. The similarity is made based on a specific "key" (because the test only makes sense when performed on ordered collections). 1. Added the `tableDiff` test definition. 2. Implemented a "runtime" parameters feature which injects additional parameters for the test at runtime. 3. Integration tests (because of course). This feature was not tested end-to-end yet because "array" data * pydantic v2 * format * format * format and added data diff to setup.py * format * fixed param issue which has type ARRAY * fixed runtime_parameter_setter * moved models to parent directory * handle errors in table diff * fixed issue with edit test case * format * added more details to pytest skip * format * refactor: Improve createTestCaseParameters function in DataQualityUtils * fixed unit test * removed unused fixture * removed validator.py * fixed tests * added validate kwarg to tests_mixin * removed "postgres" data diff extra as they interfere with psycopg2-binary * fixed tests * pinned tenacity for tests * reverted tenacity pinning * added ui support for test diff * fixed dq cypress and added edit flow * organized the test case * added dialect support * fixed tests * option style fix * fixed calculation for passing/failing rows * restrict the tableDiff test to limited services * set where to None if blank string * fixed where clause * fixed tests for where clause * use displayName in place of name in edit form * added docs for RuntimeParameterSetter * fixed cypress --------- Co-authored-by: Shailesh Parmar <shailesh.parmar.webdev@gmail.com>
2024-06-20 16:54:12 +02:00
keyColumns: List[str]
extraColumns: List[str]
whereClause: Optional[str]
[GEN-1996] feat(data-quality): use sampling config in data diff (#18532) * feat(data-quality): use sampling config in data diff - get the table profiling config - use hashing to sample deterministically the same ids from each table - use dirty-equals to assert results of stochastic processes * - reverted missing md5 - added missing database service type * - use a custom substr sql function * fixed nounce * added failure for mssql with sampling because it requires a larger change in the data-diff library * fixed unit tests * updated range for sampling
2024-11-11 10:07:23 +01:00
table_profile_config: Optional[TableProfilerConfig]
Reference in New Issue Copy Permalink