* Allowing to trigger an app with a payload
* added option to pass custom payload to triggered job
* feat(apps): add custom config to trigger
- added custom config for app triggers
* added handling for configuration validation on apps
* - removed `oneOf` from app configuration scheme
- generated ts files for apps
* added CONFIG_OVERRIDE_KEY
* format
* fixed app config key
* format
* format
* fixed python app runner to work with new schema
---------
Co-authored-by: sushi30 <imri.paran@gmail.com>
* feat(app): add config to status
add config to the reported status of the ingestion pipeline
* added separate pipeline service client call for external apps
* fix masking of pydantic model
* - overload model_dump to mask secrets instead of a separate method
- moved tests to test_custom_pydantic.py
* fix: execution time
* fix: mask secrets in dump json
* fix: for python3.8
* fix: for python3.8
* fix: use mask_secrets=False when dumping a model for create
* format
* fix: update mask_secrets=False for workflow configurations
* fix: use context directly when using model_dump_json
* fix: default behavior when dumping json
* format
* fixed tests
* backend
* format & tests
* rename backend
* migrations and ingestion
* format & tests
* format & tests
* tests
* format & tests
* tests
* updated ui side of changes
* addressing comment
* fixed failing unit test
* fix test list
* added e2e test, and fixed existing test
---------
Co-authored-by: Shailesh Parmar <shailesh.parmar.webdev@gmail.com>
* ref(profiler): use di for system profile
- use source classes that can be overridden in system profiles
- use a manifest class instead of factory to specify which class to resolve for connectors
- example usage can be seen in redshift and snowflake
* - added manifests for all custom profilers
- used super() dependency injection in order for system metrics source
- formatting
* - implement spec for all source types
- added docs for the new specification
- added some pylint ignores in the importer module
* remove TYPE_CHECKING in core.py
* - deleted valuedispatch function
- deleted get_system_metrics_by_dialect
- implemented BigQueryProfiler with a system metrics source
- moved import_source_class to BaseSpec
* - removed tests related to the profiler factory
* - reverted start_time
- removed DML_STAT_TO_DML_STATEMENT_MAPPING
- removed unused logger
* - reverted start_time
- removed DML_STAT_TO_DML_STATEMENT_MAPPING
- removed unused logger
* fixed tests
* format
* bigquery system profile e2e tests
* fixed module docstring
* - removed import_side_effects from redshift. we still use it in postgres for the orm conversion maps.
- removed leftover methods
* - tests for BaseSpec
- moved get_class_path to importer
* - moved constructors around to get rid of useless kwargs
* - changed test_system_metric
* - added linage and usage to service_spec
- fixed postgres native lineage test
* add comments on collaborative constructors
* support side effects on source classes by always importing source class
* streamlined error message
* fixed service type extraction for test suite pipeline
* - replaced "custom" with constant
- added quotes for the plugin exception for copy/paste ergonomics
* Refactor output_handlers to a WorkflowOutputHandler class
* Add old methods as deprecated to avoid breaking changes
* Extract WorkflowInitErrorHandler from workflow_output_handler
* Fix static checks
* Fix tests
* Fix tests
* Update code based on comments from PR
* Update comment
* added trino integration test
* - removed warnings for classes which are not real tests
- removed "helpers" as its being used
* use a docker network instead of host
* print logs for hive failure
* removed superset unit tests
* try pinning requests for test
* try pinning requests for test
* wait for hive to be ready
* fix trino fixture
* - reduced testcontainers_config.max_tries to 5
- remove intermediate containers
* print with logs
* disable capture logging
* updated db host
* removed debug stuff
* removed debug stuff
* removed version pin for requests
* reverted superset
* ignore trino integration on python 3.8
* feat(nosql-profiler): row count
1. Implemented the NoSQLProfilerInterface as an entrypoint for the nosql profiler.
2. Added the NoSQLMetric as an abstract class.
3. Implemented the interface for the MongoDB database source.
4. Implemented an e2e test using testcontainers.
* added profiler support for mongodb connection
* doc
* use int_admin_ometa in test setup
* - fixed linting issue in gx
- removed unused inheritance
* moved the nosql function into the metric class
* feat(profiler): add dynamodb row count
* feat(profiler): add dynamodb row count
* formatting
* validate_compose: raise exception for bad status code.
* fixed import
* format
* feat(nosql-profiler): added sample data
1. Implemented the NoSQL sampler.
2. Some naming changes to the NoSQL adaptor to avoid fixing names with the profiler interface.
3. Tests.
* added default sample limit
* formatting
* fixed import
* feat(profiler): dynamodb sample data
* tests for dynamo db sample data
* format
* format
* use service connection for nosql adaptor factory
* fixed tests
* format
* fixed after merge
* Implement Initial MultiThread suggestion
* Update all the ingestion sources to use the new ContextManager
* Fix missing wraps on decorator
* Fix Unittests
* Fix linters
* Fix linters
* Fix BigQuery UnitTests
* Add UnitTests to the newly created code
* Fix unittest
* change the threads from table to schemas
* Update README.md
* Small change suggested by Sonar
* Slight change to test a different way to multithread over tables
* Debug changes
* More multithread tests
* Remove uneeded wait time
* Testing
* refactor code based on removal of time.sleep
* Fix wrong paste
* Improve ExecutionTimeContextManager
* Fix missing .get() and unit tests
* Fix conflicting changes
* Update Multithread logic with the incremental extraction
* Fix linters
* Fix unittest
* Remove commented code
* Fix Unittests
* Fix checkstyle
* Change default to threads = 1
* linting: fix python linting
* fix: get column types from parquet schema for parquet files
* style: python linting
* fix: remove displayType check in test as variation depending on OS
* Fix#14786: Suggestions API
* Handle suggestions in ometa
* Minor: Optimise Databricks Client (#14776)
* MINOR - Fix SP topology context & Looker usage context (#14816)
* MINOR - Fix SP topology context & Looker usage context
* MINOR - Fix SP topology context & Looker usage context
* Fix tests
* Fixes#14598: Fix Tags / Labels ingestion on includeTags as False (#14782)
* fix(ui): password error message for char limits (#14808)
* fix(ui): password error message for char limits
* fix java side code
* Fixes#13556: Support for Salesforce table description ingestion (#14733)
* ISSUE-13556: Add suport for Salesforce table description ingestion
* ISSUE-13556: Remove unnecessary blank line
* ISSUE-13556: Fix to get description for each table
---------
Co-authored-by: Teddy <teddy.crepineau@gmail.com>
* MINOR - Better handling of Ingestion Pipeline Status (#14792)
* MINOR - Better handling of Ingestion Pipeline Status
* format
* format
* MINOR: Added table validation for cost analysis data (#14793)
* Added validation for cost analysis source
* centralized life cycle logic
* CYPRESS: simplify side navigation click in cypress (#14818)
* simplify side navigation click in cypress
* make sidbar item uses common enum
* fix cypress failure of outside import
* fix(#14326): tier dropdown is not working in advance search (#14780)
* improvement in advance search based on custom property
* fix a reading undefined property issue
* wip: advance search based on tier
* some code cleanup and improvement
* some fixes
* fix: ui flicker when advanceSearched is apply and refresh the page
* some cleanup
* no need to call customproperty api call, if entity not suppport customProperties
* minor change
* fix: autocomplete not working in tier search option in advance search modal
* added unit test for advance search provider component
* some cleanup
* added testcase for open modal
* added testcase for resetAllFilters method
* removed unwanted code
* added e2e test for testing tier advance search
* fix: e2e search flow for single field
* fix: string field not working after giving listValues in TierSearch
* fix: group query e2e test fix
* used asyncFetch way to get the tierOptions synchronously
* some cleanup
* remove unwanted lines
* some cleanup
* fix: selected option show option value instead of option title
* fix(minor): update skip icon for executions (#14809)
* Fixes#14803: ignore capitalization when confirming deletes (#14804)
* ignore case when confirming deletes
* Test confirmation of deletes works when case differs
Added test case for 'delete' as the confirmation text.
* minor(config): update openmetadata-ui code reviewers (#14823)
* Add Tests
* Add list/accept/reject apis
* initial ui changes
* localisation
* show suggestion for empty description
* ui feedbacks
* Fix permission check for entities without owner
* Fix entityLink and add tests
* Add update suggestion WIP
* Fix test
* Fix PUT and Pagination
* Fix styling
* update test
* Update status
* add OM server connection in apps
* add permissions check
* Fix CI
* Remove TODO
* Fix feedResourceTest
* fix unit tests
* add private configs for apps
* add private configs for apps
* fix update application icons
* minor center align icon
* add private configs for apps
* Format
* Fix pydantic gen
* Remove token
* Update name
* Rework private conf
* Fix apps
* Fix apps
* Format
* Format
* show metapilot only if its installed
---------
Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
Co-authored-by: Mayur Singal <39544459+ulixius9@users.noreply.github.com>
Co-authored-by: Ayush Shah <ayush@getcollate.io>
Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
Co-authored-by: kwgdaig <18678754+kwgdaig@users.noreply.github.com>
Co-authored-by: Teddy <teddy.crepineau@gmail.com>
Co-authored-by: Onkar Ravgan <onkar.10r@gmail.com>
Co-authored-by: Ashish Gupta <ashish@getcollate.io>
Co-authored-by: Abhishek Porwal <80886271+Abhishek332@users.noreply.github.com>
Co-authored-by: Carlo Q <carlo@machina.bio>
Co-authored-by: karanh37 <karanh37@gmail.com>
* DQ BaseWorkflow
* Test suite runner
* test Suite workflow
* Refactor DQ for BaseWorkflow
* Lint
* Fix source
* Fix source
* Fix source
* Fix source
* Fix test
* Fix test
* Fix test
* feat: updated DI workflow to inherit from BaseWorkflow + split processor and producer classes
* feat: __init__.py files creation
* feat: updated workflow import classes in code and doc
* feat: moved kpi runner from runner to processor folder
* fix: skip failure on list entities
* feat: deleted unused files
* feat: updated status reporter
* feat: ran linting
* feat: fix test error with typing and fqn
* feat: updated test dependencies
* feat: ran linting
* feat: move execution order up
* feat: updated cost analysis report to align with new workflow
* feat: fix entity already exists for pipeline entity status
* feat: ran python linting
* feat: move skip_on_failure to method
* feat: ran linting
---------
Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
