* wip
* feat: trigger external apps with override config
- Added in openmetadata-airflow-apis functionality to trigger DAG with feature.
- Modified openmetadata-airflow-apis application runner to accept override config from params.
- Added overloaded runPipeline with `Map<String,Object> config` to allow triggering apps with configuration. We might want to expand this to all ingestion pipelines. For now its just for apps.
- Implemented an example external app that can be used to test functionality of external apps. The app can be enabled by setting the `ENABLE_APP_HelloPipelines=true` environment variable.
* fix class doc for application
* fixed README for airflow apis
* fixes
* set HelloPipelines to disabeld by default
* fixed basedpywright errros
* fixed app schema
* reduced airflow client runPipeline to an overload with null config
removed duplicate call to runPipeline in AppResource
* Update openmetadata-docs/content/v1.7.x-SNAPSHOT/developers/applications/index.md
Co-authored-by: Matias Puerta <matias@getcollate.io>
* deleted documentation file
---------
Co-authored-by: Matias Puerta <matias@getcollate.io>
* feat(app): add config to status
add config to the reported status of the ingestion pipeline
* added separate pipeline service client call for external apps
* fix masking of pydantic model
* - overload model_dump to mask secrets instead of a separate method
- moved tests to test_custom_pydantic.py
* fix: execution time
* fix: mask secrets in dump json
* fix: for python3.8
* fix: for python3.8
* fix: use mask_secrets=False when dumping a model for create
* format
* fix: update mask_secrets=False for workflow configurations
* fix: use context directly when using model_dump_json
* fix: default behavior when dumping json
* format
* fixed tests
* configure api service metadata
* add rest api service
* fix test con. pyformat changes
* add models, fix test con.
* improve test con.
* add docs, side doc
* fix model data parse, url error fix
* add tests
* fix pytest errors
---------
Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
* Add multiple owners
* Multi Ownership
* Issue #17012: Multi User/Team Ownership
* Issue #17012: Multi User/Team Ownership
* Issue #17012: Multi User/Team Ownership - Fix Tests - Part 1
* Issue #17012: Multi User/Team Ownership - Fix Tests - Part 2
* Issue #17012: Multi User/Team Ownership - Fix Tests - Part 3
* Issue #17012: Multi User/Team Ownership - Fix Tests - Part 4
* Issue #17012: Multi User/Team Ownership - Fix Tests - Part 5
* Issue #17012: Multi User/Team Ownership - Fix Tests - Part 6
* Issue #17012: Multi User/Team Ownership - Fix Tests - Part 7
* Issue #17012: Multi User/Team Ownership - Fix Tests - Part 8
* Add Migrations for Owner Thread
* update ingestion for multi owner
* fix pytests
* fixed checkstyle
* Add Alert Name to Publishers (#17108)
* Add Alert Name to Publishers
* Fix Test
* Add Bound to Setuptools (#17105)
* Minor: fixed testSummaryGraph issue (#17115)
* feat: updated multi pipeline ui as per new mock (#17106)
* feat: updated multi pipeline ui as per new mock
* translation sync
* fixed failing unit test
* fixed playwright test
* fixed viewService click issue
* sorted pipeline based on test case length
* Added domo federated dataset support (#17061)
* fix usernames (#17122)
* Doc: Updated Doris & Redshift Docs (#17123)
Co-authored-by: Prajwal Pandit <prajwalpandit@Prajwals-MacBook-Air.local>
* Fix#12677: Added Synapse Connector - docs and side docs (#17041)
* Fix#17098: Fixed case sensitive partition column name in Bigquery (#17104)
* Fixed case sensitive partiion col name bigquery
* update test
* #13876: change placement of comment and close button in task approval workflow (#17044)
* change placment of comment and close button in task approval workflow
* minor change
* playwright test for the close and comment function
* supported ref in activityFeedEditor
* fix playwright test
* added playwright test for data steward
* fix the test for the data streward user
* fix the close button not showing if task has no suggestions and icon fixes
* fix sonar issue
* change glossary and add suggestion button to dropdown button
* fix the glossary failure due to button change
* icon change for add tag and description
* fix glossary cypress failure due to button chnages
* changes as per comments
* MINOR: docs links fix (#17125)
* alation link fix
* dbt yaml config source link fix
* bigquery doc fix
* Explore tree feedbacks (#17078)
* fix explore design
* update switcher icon
* show menu when search query exists
* fix selection of active service
* fix type error
* fix tests
* fix tests
* fix tests
* MINOR: Databricks view TableType fix (#17124)
* Minor: fixed AUT test (#17128)
* Fix#16692: Override Lineage Support for View & Dashboard Lineage (#17064)
* #17065: fix the tags not rendering in selector after selection in edit tags task (#17107)
* fix the tags not rendering in selector after selection in edit tags taks
* added playwright test
* minor changes
* minor fix
* fix the tags not updating in edit and accept tag
* fix explore type changes for collate (#17131)
* MINOR: changed log level to debug (#17126)
* changed log level to debug
* fixed type
* changed type to optional
* Get feed and count data of soft deleted user (#17135)
* Doc: Adding OIDC Docs (#17139)
Co-authored-by: Prajwal Pandit <prajwalpandit@Prajwals-MacBook-Air.local>
* Doc: Updating Profiler Workflow Docs URL (#17140)
Co-authored-by: Prajwal Pandit <prajwalpandit@Prajwals-MacBook-Air.local>
* fix playwright and cypress (#17138)
* Minor: fixed edit modal issue for sql test case (#17132)
* Minor: fixed edit modal issue for sql test case
* fixed test
* Minor: Added whats new content for 1.4.6 release (#17148)
* MINOR [GEN-799]: add option to disable manual trigger using scheduleType (#17031)
* fix: raise for triggering system app
* added scheduleType ScheduledOrManual
* minor: remove "service" field from required properties in createAPIEndpoint schema (#17147)
* initial commit multi ownership
* update glossary and other entities
* update owners
* fix version pages
* fix tests
* Update entity_extension to move owner to array (#17200)
* fix tests
* fix api page errors
* fix owner label design
* locales
* fix owners in elastic search source
* fix types
* fix tests
* fix tests
* Updated CustomMetric owner to entityReferenceList. (#17211)
* Fix owners field in search mappings
* fix search aggregates
* fix inherited label
* Issue #17012: Multi User/Team Ownership - Fix Tests - Part 9
* Fix QUeries
* Fix Mysql Queries
* Typo
* fix tests
* fix tests
* fix tests
* fix advanced search constants
* fix service ingestion tests
* fix tests
---------
Co-authored-by: mohitdeuex <mohit.y@deuexsolutions.com>
Co-authored-by: Onkar Ravgan <onkar.10r@gmail.com>
Co-authored-by: Mohit Yadav <105265192+mohityadav766@users.noreply.github.com>
Co-authored-by: Ayush Shah <ayush@getcollate.io>
Co-authored-by: Shailesh Parmar <shailesh.parmar.webdev@gmail.com>
Co-authored-by: k.nakagaki <141020064+nakaken-churadata@users.noreply.github.com>
Co-authored-by: Prajwal214 <167504578+Prajwal214@users.noreply.github.com>
Co-authored-by: Prajwal Pandit <prajwalpandit@Prajwals-MacBook-Air.local>
Co-authored-by: Suman Maharana <sumanmaharana786@gmail.com>
Co-authored-by: Ashish Gupta <ashish@getcollate.io>
Co-authored-by: harshsoni2024 <64592571+harshsoni2024@users.noreply.github.com>
Co-authored-by: Karan Hotchandani <33024356+karanh37@users.noreply.github.com>
Co-authored-by: Mayur Singal <39544459+ulixius9@users.noreply.github.com>
Co-authored-by: Imri Paran <imri.paran@gmail.com>
Co-authored-by: sonika-shah <58761340+sonika-shah@users.noreply.github.com>
Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com>
Co-authored-by: karanh37 <karanh37@gmail.com>
Co-authored-by: Siddhant <86899184+Siddhanttimeline@users.noreply.github.com>
* Refactor output_handlers to a WorkflowOutputHandler class
* Add old methods as deprecated to avoid breaking changes
* Extract WorkflowInitErrorHandler from workflow_output_handler
* Fix static checks
* Fix tests
* Fix tests
* Update code based on comments from PR
* Update comment
* encrypt jwt
* encrypt jwt
* Handle token revokes with SM
* #10192 - Validate both include and excludes filters
* #10192 - Validate both include and excludes filters
* docs
* linting: fix python linting
* fix: get column types from parquet schema for parquet files
* style: python linting
* fix: remove displayType check in test as variation depending on OS
* Fix#14786: Suggestions API
* Handle suggestions in ometa
* Minor: Optimise Databricks Client (#14776)
* MINOR - Fix SP topology context & Looker usage context (#14816)
* MINOR - Fix SP topology context & Looker usage context
* MINOR - Fix SP topology context & Looker usage context
* Fix tests
* Fixes#14598: Fix Tags / Labels ingestion on includeTags as False (#14782)
* fix(ui): password error message for char limits (#14808)
* fix(ui): password error message for char limits
* fix java side code
* Fixes#13556: Support for Salesforce table description ingestion (#14733)
* ISSUE-13556: Add suport for Salesforce table description ingestion
* ISSUE-13556: Remove unnecessary blank line
* ISSUE-13556: Fix to get description for each table
---------
Co-authored-by: Teddy <teddy.crepineau@gmail.com>
* MINOR - Better handling of Ingestion Pipeline Status (#14792)
* MINOR - Better handling of Ingestion Pipeline Status
* format
* format
* MINOR: Added table validation for cost analysis data (#14793)
* Added validation for cost analysis source
* centralized life cycle logic
* CYPRESS: simplify side navigation click in cypress (#14818)
* simplify side navigation click in cypress
* make sidbar item uses common enum
* fix cypress failure of outside import
* fix(#14326): tier dropdown is not working in advance search (#14780)
* improvement in advance search based on custom property
* fix a reading undefined property issue
* wip: advance search based on tier
* some code cleanup and improvement
* some fixes
* fix: ui flicker when advanceSearched is apply and refresh the page
* some cleanup
* no need to call customproperty api call, if entity not suppport customProperties
* minor change
* fix: autocomplete not working in tier search option in advance search modal
* added unit test for advance search provider component
* some cleanup
* added testcase for open modal
* added testcase for resetAllFilters method
* removed unwanted code
* added e2e test for testing tier advance search
* fix: e2e search flow for single field
* fix: string field not working after giving listValues in TierSearch
* fix: group query e2e test fix
* used asyncFetch way to get the tierOptions synchronously
* some cleanup
* remove unwanted lines
* some cleanup
* fix: selected option show option value instead of option title
* fix(minor): update skip icon for executions (#14809)
* Fixes#14803: ignore capitalization when confirming deletes (#14804)
* ignore case when confirming deletes
* Test confirmation of deletes works when case differs
Added test case for 'delete' as the confirmation text.
* minor(config): update openmetadata-ui code reviewers (#14823)
* Add Tests
* Add list/accept/reject apis
* initial ui changes
* localisation
* show suggestion for empty description
* ui feedbacks
* Fix permission check for entities without owner
* Fix entityLink and add tests
* Add update suggestion WIP
* Fix test
* Fix PUT and Pagination
* Fix styling
* update test
* Update status
* add OM server connection in apps
* add permissions check
* Fix CI
* Remove TODO
* Fix feedResourceTest
* fix unit tests
* add private configs for apps
* add private configs for apps
* fix update application icons
* minor center align icon
* add private configs for apps
* Format
* Fix pydantic gen
* Remove token
* Update name
* Rework private conf
* Fix apps
* Fix apps
* Format
* Format
* show metapilot only if its installed
---------
Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
Co-authored-by: Mayur Singal <39544459+ulixius9@users.noreply.github.com>
Co-authored-by: Ayush Shah <ayush@getcollate.io>
Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
Co-authored-by: kwgdaig <18678754+kwgdaig@users.noreply.github.com>
Co-authored-by: Teddy <teddy.crepineau@gmail.com>
Co-authored-by: Onkar Ravgan <onkar.10r@gmail.com>
Co-authored-by: Ashish Gupta <ashish@getcollate.io>
Co-authored-by: Abhishek Porwal <80886271+Abhishek332@users.noreply.github.com>
Co-authored-by: Carlo Q <carlo@machina.bio>
Co-authored-by: karanh37 <karanh37@gmail.com>
* DQ BaseWorkflow
* Test suite runner
* test Suite workflow
* Refactor DQ for BaseWorkflow
* Lint
* Fix source
* Fix source
* Fix source
* Fix source
* Fix test
* Fix test
* Fix test
* feat: updated DI workflow to inherit from BaseWorkflow + split processor and producer classes
* feat: __init__.py files creation
* feat: updated workflow import classes in code and doc
* feat: moved kpi runner from runner to processor folder
* fix: skip failure on list entities
* feat: deleted unused files
* feat: updated status reporter
* feat: ran linting
* feat: fix test error with typing and fqn
* feat: updated test dependencies
* feat: ran linting
* feat: move execution order up
* feat: updated cost analysis report to align with new workflow
* feat: fix entity already exists for pipeline entity status
* feat: ran python linting
* feat: move skip_on_failure to method
* feat: ran linting
---------
Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
* Reverted the ES config step for Data Insights pipeline
* es sink fixes for ingestion
---------
Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
Co-authored-by: ulixius9 <mayursingal9@gmail.com>
* Add fine-grained health validation for Airflow APIs
* Add ingestion version response
* Improve messaging
* Format
* Format
* Update response
* ui:update the hook to return airflow status based on response status field
* ui: add unit test for useAirflowStatus hook
* chore: only show test connection if airflow is available
* feat: add airflow message banner
* chore: update icon and background color
* chore: update typography to text
* test: add unit test
* address comments
* chore: show banner on service detail page
* fix: update test suite api workflow to reflect new implementation
---------
Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com>
Co-authored-by: Teddy Crepineau <teddy.crepineau@gmail.com>
* Deprecate location and old storage service
* Format
* Fix test
* Refactor
* Clean location
* Rename object store to storage
* Rename object store to storage
* Rename object store to storage
* Format
* Format
* Refactor object store for storage
* Refactor object store for storage
* Rename object store to storage
* Fix test
* Fix test
* Format
* chore(ui): change Objectstore to Storage
* Fixes
* Fix test
* Remove storage service from Glue cypress
---------
Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com>
