* Mask sensitive info from API responses
* Rename converter classes
* Add missing Java classes from JSON schemas and class converters
* Update test service connection schema
* Update datalakeConnection JSON schema and fix some tests
* Fix AlertsRuleEvaluatorResourceTest and minor error in run_local_docker.sh
* Fix Pipeline and Database service tests
* Minor refactor
* Fix CsvUtilTest
* Fix EventMonitorFactoryTest
* Fix CloudWatchEventMonitorTest
* Update datalake metadata
* Update bigquery metadata
* Fix test connection functionality
* Fix OMeta service api test
* Update gcsValues title and revert changes in GH actions
* Mask sensitive enabled by default for local docker
* Add missing tests
* Address PR comments
* Address PR comments
* fix ui breaks on gcsValues.json
* Address PR comments
* Minor refactor
---------
Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
* WIP: pii tagging using spacy
* added test cases and changes as per comment
* fix python checkstyle
* fix python checkstyle
* added score, test_cases and docs update
* solved merge conflict
* fix python checkstyle
* remove pii tagging using regex
* fix python test
* lib changes and added some test case
* changed as per comment
* fix: python test
* fix: changes to get source_config
* fix: changes as per comment
* fix(dataInsight): handle cases where entity would be deleted from OM
* Update ingestion/src/metadata/data_insight/processor/web_analytic_report_data_processor.py
Added explanation in code comments
Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
* fix(dataInsight): tests failure
---------
Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
* feat(profiler): renamed module to
* feat(profiler): added dbt-artifacts-parser to test setup.py
* feat(profiler): refactor workflow and interface
* feat(profiler): linting
* feat(profiler): removed old profiler modules
* feat(profiler): added support for value and integer range partition
* feat(profiler): fixed linting
* feat(profiler): added partitionning support for datalake profiler
* feat(profiler): removed `ProfilerInterfaceArgs` class
* feat(profiler): address comments
* feat(profiler): Added `OTHER` as an `IntervalType` for UI type generation
* feat(testSuite): extracted out column test for SQA type
* refactor(testSuite): extracted SQA column and table tests into their own classes
* refactor(testSuite): Added pkutil namespace package style for test suite classes
* refactor(testSuite): added dynamic importer function for test cases
* refactor(testSuite): black formatting
* refactor(testSuite): fixed linting issues
* refactor(testSuite): refactor metrics for dataframe
* refactor(testSuite): Added Mixins and base methods
* refactor(testSuite): extrcated out get bound for floats
* refactor(testSuite): Added pandas column test cases
* refactor(testSuite): Deleted old column tests
* refactor(testSuite): Added table tests for datalake
* refactor(testSuite): Removed old tests definition
* refactor(testSuite): changed registry to dynamic class inport
* refactor(testSuite): renamed dl_fn to df_fn
* refactor(testSuite): updated registry unit test
* refactor(testSuite): updated import path to sqa like column
* refactor(testSuite): cleaned up imports in old files
* refactor(testSuite): harmonzied SQALikeColumn object to replicate SQA Column object
* refactor(testSuite): linting
* refactor(testSuite): linting
* refactor(testSuite): raise expection on DQ exception
* refactor(testSuite): linting
* refactor(testSuite): removed pandas from base requirements
* refactor(testSuite): Added __futur__ for py3.7 type hint
* refactor(testSuite): added `df` to good-names
* refactor(testSuite): renamed Handler to Validator
* refactor(testSuite): Added test inheritance for column tests
* refactor(testSuite): cleaned up column type check
* refactor(testSuite): cleaned up typo
* refactor(testSuite): extracted main table test logic into parent class
* refactor(testSuite): linting
* refactor(testSuite): linting fixes
* refactor(testSuite): address doc string and linting issues
* Change entityReference to entity name or fullyQualifiedName
* Change backend code and tests to use FQN
* UI change for using fqns instead of EntityReference
* Ingestion framework changes for using fqns instead of EntityReference
* Fix test failures
* Fixed python tests and sample data new
* fix: minor ui changes for fqn
* Fixed python integration tests
* Fixed superset tests
* fix UI tests
* fix type issue
* fix cypress
* fix name for testcase
---------
Co-authored-by: Onkar Ravgan <onkar.10r@gmail.com>
Co-authored-by: karanh37 <karanh37@gmail.com>
Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
* Fix: added changes for pii sensitive
* Fix: removed comments
* Fix: python checkstyle
* differtiate between sensitive and non sensitive tag
* fix: python test
* fix: added tests
* fix: maven CI
* Update airflow loggers and rename ometa loggers
* ANSI print to logger
* Remove colored logging from tests
* Merge ometa_logger into the one used in loggers class
* linting
* linting
Co-authored-by: Nahuel Verdugo Revigliono <nahuel@getcollate.io>
* refactor(profiler): integrated getter func.
Removed metric getter function from their own file.
Added metric getter to their own interface classs.
created dispatch by value methdo to dispatch metric getter func.
* feature(profiler): added systemProfiler schema
* feat(profiler): workflow fresh. & snflk impl.
* feat(profiler): freshness endpoint for put and get
* feat(profiler): added system met. for redshift
* feat(profiler): freshness met. for bigquery
* fix(profiler): keyword not found in func
* feat(profiler): Added sample data for freshness
* fix(profiler): fetch previous day for BQ
* fix(profiler): sonar + data fetching logic
* fix: typo in SystemMetric Class
* fix: linting
* fix: extracted out EntityList class into models.py
* Mapping from kpi to dichart(One to one)
* Mapping from kpi to dichart(One to one)
* Removed second tests creating KPI
* Fix some ui improve as well
* review comment changed function name
Co-authored-by: Teddy Crepineau <teddy.crepineau@gmail.com>
* Removed part of the secrets manager implementation in the server side
* Removed part of the secrets manager implementation in the openmetadata python library side
* Remove deprecated test
* Address pylint checks
* Address new pylint checks
* Address PR comments
* Fix import on airflows apis
* Clear singleton instances for running Ometa secrets manager test
* Moved webanalytics type in its own folder
* Added data insight chart api endpoint
* Jave formatting
* Added resource descriptor
* Added metadata entity endpoint
* Added aggregation endpoint for dataInsight
* Fix tag name
* Added logic to ingestion pipeline resource to add ES config info if pipeline type is dataInsight
* added domo to test subpackage
* cleaned up branch by removing commit from issue-8353 that were not merged in main
* Added web analytics data refinement
* Added get_status function
* Added from __futur__ for typing
* Fixed typos brought up during reviews
* added data insight module to refine and ingest data
* Added json schema for data insight workflow
* Added mixin for data insight
* Added tests for data insight workflow and methods
* Fixed python style
* Fixed python linting
* Fixed unit test
* Added data insight worflow + workflow logging
* Fix python style
* Renamed analytic -> data_insight + added test to make + fixed style
* Fixed java style
* Fixed code smells + python style
* Ignore ES linting for now
* Fixed comments from review
* fix python formatting
* Fixe PipelineType attribute for data insight
* Clean up test suite workflow and interface
* Fixed tests
* Split profiler and testSuite interfaces
* Cleaned up workflows and runners
* Fixed code formatting
* - remove old code
- remove `table` attribute used for testing and used mock instead
* Fixed execution bugs from refactor
* Fixed static type checking for profiler/api/workflow.py
* Fixed linting
* Added __init__ files
* Update GE logic to the new test API
* Cleaned up part of the code from
update to TestSuiteMixin
* Added + fixed tests
* Deleted unused files for GE tests
* fixed python tests
* Added database filter in workflow
* Removed association between profiler and data quality
* fixed tests with removed association
* Fixed sonar code smells and bugs
* Updated profiler workflow to:
- support only running profiler (removed test run)
- support column inclusion and exclusion
- added back support for partitioned table and sample
* moved status to workflow
* Fixed tests
* removed test logic from profiler sink
* Added logic to return sample from workflow sample value
* Added profiler examples
* Updated documentation for profiler
* Fixed code smells
* commited changed to profiler
* initial commit of the revamp workflow
* Fixed python formating
* cleaned up profiler submodule by removing test related files and functions
* Added airflow DAG logic for testSuite workflow
* Fixed code smells + added airflow ingestion tests + fixed comments
