* fix: sqa table reference
* style: ran python linting
* fix: added raw dataset to query runner
* fix: get table and schema name from orm object
* fix: get table level config for table tests
* add dbt freshness check
* docs
* run linting
* add test case param definition
* fix test case param definition
* add config for dbt http, fix linting
* refactor (only create freshness test definition when user executed one)
* fix dbt files class
* fix dbt files class 2
* fix dbt objects class
* fix linting
* fix pylint
* fix linting once and for all
---------
Co-authored-by: Teddy <teddy.crepineau@gmail.com>
* ref(data-quality): modularized test case validator import
- removed test_suite_factory
- implemented TestCaseImporter
- removed SQAValidatorBuilder and PandasValidatorBuilder in favor of a SourceType enum
- removed the orm table creation from test suite source
* format
* IValidatorBuilder -> ValidatorBuilder
* use the table from the sampler in the test suite interface
* linting
* fixed the profiler with similar solution
* removed unused inheritance
* removed unneeded super().__init__()
* removed all instances of orm_table
* fixed tests
* add reportExplicitAny=false
* fixed tests
* fix(data-diff): sampling configuration
handle the sampling condition separately for the 2 tables allowing to apply sampling on columns with mismatching cases
* format
* feat(data-quality): use sampling config in data diff
- get the table profiling config
- use hashing to sample deterministically the same ids from each table
- use dirty-equals to assert results of stochastic processes
* - reverted missing md5
- added missing database service type
* - use a custom substr sql function
* fixed nounce
* added failure for mssql with sampling because it requires a larger change in the data-diff library
* fixed unit tests
* updated range for sampling
* feat(statistics-profiler): use statistics tables to profile trino tables
- implemented the collaborative root class
- added the "useStatistics" profiler parameter
- added the "supportsStatistics" database connection property
- implemented the ProfilerWithStatistics and StoredStatisticsSource to add this functionality to specific profilers
- implemented TrinoStoredStatisticsSource for specific trino statistics logic
* added ABC to terminal classes in collaborative root
* fixed docstring for TestSuiteInterface
* reverted unintended changes
* typo
* feat: added column value to be in expected location test
* fix: renamed value -> values
* doc: added 1.6 documentatio entry
* style: ran python linting
* fix: move data packaging to pyproject.yaml
* fix: add init file back for data package
* fix: failing test case
* Add flake.nix
* Add lockfile for flake
* Update nix environment and document usage
* Add schema for exasol connector
* Add Exasol definitions to databaseService
* Fix error in exasol connector schema
* Add additional connection options/settings to exasol connector
* Add exasol-connector to ui
* Add depdencies for exasol-connector
* Update notes
* Update ingestion code
* Add Basic Documentation for Exasol Connector
* Update flake file
* Add developer notes
* Add python script which can be used as entry point for debugging in ide
* Add config file which can be used for debugging (manual execution)
* Update debug script
* Update developer notes
* Remove old developer notes
* Add .venv to gitignore
* Update dev notes
* Update development notes
* Update ExasolSource
* Establish basic connection to Exasol DB from connector
* Update exasol connector connection settings
* Add service_spec for exasol plugin
* Remove development files
* Remove unused module
* Applied code formatter
* Update exasol dependency constraint(s)
* Add unit test for exasol connection url(s)
* Fixed test expectations for exasol connection url test(s)
* Adjust the test query for the Exasol connection test
* ref(profiler): use di for system profile
- use source classes that can be overridden in system profiles
- use a manifest class instead of factory to specify which class to resolve for connectors
- example usage can be seen in redshift and snowflake
* - added manifests for all custom profilers
- used super() dependency injection in order for system metrics source
- formatting
* - implement spec for all source types
- added docs for the new specification
- added some pylint ignores in the importer module
* remove TYPE_CHECKING in core.py
* - deleted valuedispatch function
- deleted get_system_metrics_by_dialect
- implemented BigQueryProfiler with a system metrics source
- moved import_source_class to BaseSpec
* - removed tests related to the profiler factory
* - reverted start_time
- removed DML_STAT_TO_DML_STATEMENT_MAPPING
- removed unused logger
* - reverted start_time
- removed DML_STAT_TO_DML_STATEMENT_MAPPING
- removed unused logger
* fixed tests
* format
* bigquery system profile e2e tests
* fixed module docstring
* - removed import_side_effects from redshift. we still use it in postgres for the orm conversion maps.
- removed leftover methods
* - tests for BaseSpec
- moved get_class_path to importer
* - moved constructors around to get rid of useless kwargs
* - changed test_system_metric
* - added linage and usage to service_spec
- fixed postgres native lineage test
* add comments on collaborative constructors
* fix(data-quality): table diff
- added handling for case-insensitive columns
- added handling for different numeric types (int/float/Decimal)
- added handling of boolean test case parameters
* add migrations for table diff
* add migrations for table diff
* removed cross type diff for now. it appears to be flaky
* fixed migrations
* use casefold() instead of lower()
* - implemented utils.get_test_case_param_value
- fixed params for case sensitive column
* handle bool test case parameters
* format
* testing
* format
* list -> List
* list -> List
* - change caseSensitiveColumns default to fase
- added migration to stay backward compatible
* - removed migration files
- updated logging message for table diff migration
* changed bool test case parameters default to always be false
* format
* docs: data diff
- added the caseSensitiveColumns parameter
requires: https://github.com/open-metadata/OpenMetadata/pull/18115
* fixed test_get_bool_test_case_param
* ref(profiler): redshift system metrics
- moved redshift system metrics to the redshift source module
- use Timestamp in data quality
- added plugin feature to test utils
* use timezone.utc
* format
* reverted unintended snowflake changes
* fixed import test_system_metrics.py
* revert
* fixed import in tests
* GEN-1322: API Entity - Remove Beta
* minor: add doc for the metadata pipeline
* api service refactor
* api service refactor backend changes
* add apiconnection in test service connection
* pytest fix
* fix java file formatting
* Fix casing of REST in ApiServiceRest.spec.ts
* Refactor REST to Rest in API classes
* minor change
* minor change
* minor change
* fix cashing for API to Api
* add playwright test for api service ingestion
* fix: playwright test
---------
Co-authored-by: harshsoni2024 <harshsoni2024@gmail.com>
* Issue-15768: Support Metric Entity
* Issue-15768: Support Metric Entity
* Issue-15768: Support Metric Entity
* Fix tests
* Fix tests
* Fix tests
* Minor: Fix tests
* ui: add metricsAPI rest utils
* ui: metric list page part 1
* feat: Add metric translations for multiple languages
* chore: Add "metric" field to SearchIndexingApplication schema
* ui: add create metric page
* ui: metric details page patch 1
* ui: add custom property and lineage support for metric entity
* ui: add expression component
* ui: add metric summary component
* chore: Update tab labels in MetricDetails and MetricVersion components
* ui: show other info like metric type, granularity, etc
* feat: Add support for metric entity in search dropdown
* feat: Rename custom property to Metric in MetricEntity.md
* feat: Add OwnerLabel component to MetricListPage
* Fix expression field in Metric
* chore: update expression to metricExpression
* ui: add metric header component with edit option
* Add metric to SearchIndexApp
* chore: Update expression to metricExpression
* ui: allow metric expression edit
* ui: update metric icon
* minor improvements
* Fix lineage indexing for Metric
* Update GlobalSettingsClassBase.ts to use MetricIcon for metrics in the global settings menu
* Fix error handling in MetricListPage component
* add related metrics
* minor improvements
* Fix relatedTerms patch
* Fix relatedTerms validation
* Add Boolean for deleted
* filter active entity from related metric list
* playwrite e2e part 1
* Refactor MetricSummary component to include RelatedMetrics in the summary panel
* test: add playwright test for metric special cases
* Add 'Metrics' to Explore Tree
* test: add e2e for add metric page
* test: add test for metric listing page content
* Add Boolean for deleted, remove deleted from suggests
* Refactor LineageProvider to handle deleted flag properly
* add playwright for metric listing
* fix test
* Add colored metric icon and update its usage in GlobalSettingsClassBase
* Fixed py_test test_ometa_endpoint for metric
---------
Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com>
Co-authored-by: mohitdeuex <mohit.y@deuexsolutions.com>
Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
Co-authored-by: SumanMaharana <sumanmaharana786@gmail.com>
Co-authored-by: Ashish Gupta <ashish@getcollate.io>
* support side effects on source classes by always importing source class
* streamlined error message
* fixed service type extraction for test suite pipeline
* - replaced "custom" with constant
- added quotes for the plugin exception for copy/paste ergonomics
* tests(datalake): use minio
1. use minio instead of moto for mimicking s3 behavior.
2. removed moto dependency as it is not compatible with aiobotocore (https://github.com/getmoto/moto/issues/7070#issuecomment-1828484982)
* - moved test_datalake_profiler_e2e.py to datalake/test_profiler
- use minio instead of moto
* fixed tests
* fixed tests
* removed default name for minio container
* configure api service metadata
* add rest api service
* fix test con. pyformat changes
* add models, fix test con.
* improve test con.
* add docs, side doc
* fix model data parse, url error fix
* add tests
* fix pytest errors
---------
Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
