Sriharsha Chintalapani 2e95fcb98d
Fix #14786: Suggestions API (#14821)
* Fix #14786: Suggestions API

* Handle suggestions in ometa

* Minor: Optimise Databricks Client (#14776)

* MINOR - Fix SP topology context & Looker usage context (#14816)

* MINOR - Fix SP topology context & Looker usage context

* MINOR - Fix SP topology context & Looker usage context

* Fix tests

* Fixes #14598: Fix Tags / Labels ingestion on includeTags as False (#14782)

* fix(ui): password error message for char limits (#14808)

* fix(ui): password error message for char limits

* fix java side code

* Fixes #13556: Support for Salesforce table description ingestion (#14733)

* ISSUE-13556: Add suport for Salesforce table description ingestion

* ISSUE-13556: Remove unnecessary blank line

* ISSUE-13556: Fix to get description for each table

---------

Co-authored-by: Teddy <teddy.crepineau@gmail.com>

* MINOR - Better handling of Ingestion Pipeline Status (#14792)

* MINOR - Better handling of Ingestion Pipeline Status

* format

* format

* MINOR: Added table validation for cost analysis data (#14793)

* Added validation for cost analysis source

* centralized life cycle logic

* CYPRESS: simplify side navigation click in cypress (#14818)

* simplify side navigation click in cypress

* make sidbar item uses common enum

* fix cypress failure of outside import

* fix(#14326): tier dropdown is not working in advance search (#14780)

* improvement in advance search based on custom property

* fix a reading undefined property issue

* wip: advance search based on tier

* some code cleanup and improvement

* some fixes

* fix: ui flicker when advanceSearched is apply and refresh the page

* some cleanup

* no need to call customproperty api call, if entity not suppport customProperties

* minor change

* fix: autocomplete not working in tier search option in advance search modal

* added unit test for advance search provider component

* some cleanup

* added testcase for open modal

* added testcase for resetAllFilters method

* removed unwanted code

* added e2e test for testing tier advance search

* fix: e2e search flow for single field

* fix: string field not working after giving listValues in TierSearch

* fix: group query e2e test fix

* used asyncFetch way to get the tierOptions synchronously

* some cleanup

* remove unwanted lines

* some cleanup

* fix: selected option show option value instead of option title

* fix(minor): update skip icon for executions (#14809)

* Fixes #14803: ignore capitalization when confirming deletes  (#14804)

* ignore case when confirming deletes

* Test confirmation of deletes works when case differs 

Added test case for 'delete' as the confirmation text.

* minor(config): update openmetadata-ui code reviewers (#14823)

* Add Tests

* Add list/accept/reject apis

* initial ui changes

* localisation

* show suggestion for empty description

* ui feedbacks

* Fix permission check for entities without owner

* Fix entityLink and add tests

* Add update suggestion WIP

* Fix test

* Fix PUT and Pagination

* Fix styling

* update test

* Update status

* add OM server connection in apps

* add permissions check

* Fix CI

* Remove TODO

* Fix feedResourceTest

* fix unit tests

* add private configs for apps

* add private configs for apps

* fix update application icons

* minor center align icon

* add private configs for apps

* Format

* Fix pydantic gen

* Remove token

* Update name

* Rework private conf

* Fix apps

* Fix apps

* Format

* Format

* show metapilot only if its installed

---------

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
Co-authored-by: Mayur Singal <39544459+ulixius9@users.noreply.github.com>
Co-authored-by: Ayush Shah <ayush@getcollate.io>
Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
Co-authored-by: kwgdaig <18678754+kwgdaig@users.noreply.github.com>
Co-authored-by: Teddy <teddy.crepineau@gmail.com>
Co-authored-by: Onkar Ravgan <onkar.10r@gmail.com>
Co-authored-by: Ashish Gupta <ashish@getcollate.io>
Co-authored-by: Abhishek Porwal <80886271+Abhishek332@users.noreply.github.com>
Co-authored-by: Carlo Q <carlo@machina.bio>
Co-authored-by: karanh37 <karanh37@gmail.com>
2024-01-31 18:51:09 -08:00

107 lines
3.6 KiB
Python

# Copyright 2021 Collate
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
# http://www.apache.org/licenses/LICENSE-2.0
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""
Generic Workflow entrypoint to execute Applications
"""
import json
from airflow import DAG
from openmetadata_managed_apis.utils.logger import set_operator_logger
from openmetadata_managed_apis.workflows.ingestion.common import (
build_dag,
build_workflow_config_property,
)
from metadata.generated.schema.entity.applications.configuration.applicationConfig import (
AppConfig,
PrivateConfig,
)
from metadata.generated.schema.entity.services.ingestionPipelines.ingestionPipeline import (
IngestionPipeline,
)
from metadata.generated.schema.metadataIngestion.application import (
OpenMetadataApplicationConfig,
)
from metadata.generated.schema.metadataIngestion.applicationPipeline import (
ApplicationPipeline,
)
from metadata.ingestion.models.encoders import show_secrets_encoder
from metadata.workflow.application import ApplicationWorkflow
from metadata.workflow.application_output_handler import print_status
def application_workflow(workflow_config: OpenMetadataApplicationConfig):
"""
Task that creates and runs the ingestion workflow.
The workflow_config gets cooked form the incoming
ingestionPipeline.
This is the callable used to create the PythonOperator
"""
set_operator_logger(workflow_config)
config = json.loads(workflow_config.json(encoder=show_secrets_encoder))
workflow = ApplicationWorkflow.create(config)
workflow.execute()
workflow.raise_from_status()
print_status(workflow)
workflow.stop()
def build_application_workflow_config(
ingestion_pipeline: IngestionPipeline,
) -> OpenMetadataApplicationConfig:
"""
Given an airflow_pipeline, prepare the workflow config JSON
"""
# Here we have an application pipeline, so the Source Config is of type ApplicationPipeline
application_pipeline_conf: ApplicationPipeline = (
ingestion_pipeline.sourceConfig.config
)
application_workflow_config = OpenMetadataApplicationConfig(
sourcePythonClass=application_pipeline_conf.sourcePythonClass,
# We pass the generic class and let each app cast the actual object
appConfig=AppConfig(
__root__=application_pipeline_conf.appConfig.__root__,
)
if application_pipeline_conf.appConfig
else None,
appPrivateConfig=PrivateConfig(
__root__=application_pipeline_conf.appPrivateConfig.__root__
)
if application_pipeline_conf.appPrivateConfig
else None,
workflowConfig=build_workflow_config_property(ingestion_pipeline),
ingestionPipelineFQN=ingestion_pipeline.fullyQualifiedName.__root__,
)
return application_workflow_config
def build_application_dag(ingestion_pipeline: IngestionPipeline) -> DAG:
"""
Build a simple metadata workflow DAG
"""
application_workflow_config = build_application_workflow_config(ingestion_pipeline)
dag = build_dag(
task_name="application_task",
ingestion_pipeline=ingestion_pipeline,
workflow_config=application_workflow_config,
workflow_fn=application_workflow,
)
return dag