Ignoring revisions in .git-blame-ignore-revs. Click here to bypass and see the normal blame view.

75 lines
2.3 KiB
Python
Raw Permalink Normal View History

# Copyright 2025 Collate
# Licensed under the Collate Community License, Version 1.0 (the "License");
[WIP] - Test Connection - Prepare the new test connection ingestion+UI logic (#10660) * Prepare the new test connection ingestion logic * Update test assert * Update Test Connection for SQA Sources * Correct return type and method doc * Handle decryption * Non SQA Database Sources * Add the run_automation script in ingestion-base * Dashboard Test Connection Changes * Pipeline, Messagin, MlModel & Metadata Sources * ui: test connect flow-1 * Unmask connection parameters before sending to Ariflow * ui: test connect flow-2 * Address review comments and pylint * pytest fix * ui: test connect flow-3 (refactoring and style fix) * ui: test connect flow-4 (fix test connection status logic) * sync local file * ui: test connect flow-5 (fix lowercase issue and styling) * ui: test connect flow-5 (show toast notifications) * test: add unit test * ui: test connect flow-5 (update service page test connection button) * Databrick fix & pytest fix * pylint * Update test * Fix merge * S3 Test connection * add style for mandatory step * sync locales * chore: add service name in workflow request * Unmask using original service connection parameters * Fix test connection unmasking * Wrap inspector function to eliminate error outside test conn * Fix linting * fix:cy test * Fix linting * address comment * refactor and fix connection type casing issue --------- Co-authored-by: ulixius9 <mayursingal9@gmail.com> Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com> Co-authored-by: Nahuel Verdugo Revigliono <nahuel@getcollate.io> Co-authored-by: Mayur Singal <39544459+ulixius9@users.noreply.github.com>
2023-03-28 06:29:13 +02:00
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
# https://github.com/open-metadata/OpenMetadata/blob/main/ingestion/LICENSE
[WIP] - Test Connection - Prepare the new test connection ingestion+UI logic (#10660) * Prepare the new test connection ingestion logic * Update test assert * Update Test Connection for SQA Sources * Correct return type and method doc * Handle decryption * Non SQA Database Sources * Add the run_automation script in ingestion-base * Dashboard Test Connection Changes * Pipeline, Messagin, MlModel & Metadata Sources * ui: test connect flow-1 * Unmask connection parameters before sending to Ariflow * ui: test connect flow-2 * Address review comments and pylint * pytest fix * ui: test connect flow-3 (refactoring and style fix) * ui: test connect flow-4 (fix test connection status logic) * sync local file * ui: test connect flow-5 (fix lowercase issue and styling) * ui: test connect flow-5 (show toast notifications) * test: add unit test * ui: test connect flow-5 (update service page test connection button) * Databrick fix & pytest fix * pylint * Update test * Fix merge * S3 Test connection * add style for mandatory step * sync locales * chore: add service name in workflow request * Unmask using original service connection parameters * Fix test connection unmasking * Wrap inspector function to eliminate error outside test conn * Fix linting * fix:cy test * Fix linting * address comment * refactor and fix connection type casing issue --------- Co-authored-by: ulixius9 <mayursingal9@gmail.com> Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com> Co-authored-by: Nahuel Verdugo Revigliono <nahuel@getcollate.io> Co-authored-by: Mayur Singal <39544459+ulixius9@users.noreply.github.com>
2023-03-28 06:29:13 +02:00
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""
Entrypoint to run an automation workflow
"""
import logging
[WIP] - Test Connection - Prepare the new test connection ingestion+UI logic (#10660) * Prepare the new test connection ingestion logic * Update test assert * Update Test Connection for SQA Sources * Correct return type and method doc * Handle decryption * Non SQA Database Sources * Add the run_automation script in ingestion-base * Dashboard Test Connection Changes * Pipeline, Messagin, MlModel & Metadata Sources * ui: test connect flow-1 * Unmask connection parameters before sending to Ariflow * ui: test connect flow-2 * Address review comments and pylint * pytest fix * ui: test connect flow-3 (refactoring and style fix) * ui: test connect flow-4 (fix test connection status logic) * sync local file * ui: test connect flow-5 (fix lowercase issue and styling) * ui: test connect flow-5 (show toast notifications) * test: add unit test * ui: test connect flow-5 (update service page test connection button) * Databrick fix & pytest fix * pylint * Update test * Fix merge * S3 Test connection * add style for mandatory step * sync locales * chore: add service name in workflow request * Unmask using original service connection parameters * Fix test connection unmasking * Wrap inspector function to eliminate error outside test conn * Fix linting * fix:cy test * Fix linting * address comment * refactor and fix connection type casing issue --------- Co-authored-by: ulixius9 <mayursingal9@gmail.com> Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com> Co-authored-by: Nahuel Verdugo Revigliono <nahuel@getcollate.io> Co-authored-by: Mayur Singal <39544459+ulixius9@users.noreply.github.com>
2023-03-28 06:29:13 +02:00
import os
import yaml
from metadata.automations.execute_runner import execute
[WIP] - Test Connection - Prepare the new test connection ingestion+UI logic (#10660) * Prepare the new test connection ingestion logic * Update test assert * Update Test Connection for SQA Sources * Correct return type and method doc * Handle decryption * Non SQA Database Sources * Add the run_automation script in ingestion-base * Dashboard Test Connection Changes * Pipeline, Messagin, MlModel & Metadata Sources * ui: test connect flow-1 * Unmask connection parameters before sending to Ariflow * ui: test connect flow-2 * Address review comments and pylint * pytest fix * ui: test connect flow-3 (refactoring and style fix) * ui: test connect flow-4 (fix test connection status logic) * sync local file * ui: test connect flow-5 (fix lowercase issue and styling) * ui: test connect flow-5 (show toast notifications) * test: add unit test * ui: test connect flow-5 (update service page test connection button) * Databrick fix & pytest fix * pylint * Update test * Fix merge * S3 Test connection * add style for mandatory step * sync locales * chore: add service name in workflow request * Unmask using original service connection parameters * Fix test connection unmasking * Wrap inspector function to eliminate error outside test conn * Fix linting * fix:cy test * Fix linting * address comment * refactor and fix connection type casing issue --------- Co-authored-by: ulixius9 <mayursingal9@gmail.com> Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com> Co-authored-by: Nahuel Verdugo Revigliono <nahuel@getcollate.io> Co-authored-by: Mayur Singal <39544459+ulixius9@users.noreply.github.com>
2023-03-28 06:29:13 +02:00
from metadata.generated.schema.entity.automations.workflow import (
Workflow as AutomationWorkflow,
)
from metadata.utils.logger import set_loggers_level
[WIP] - Test Connection - Prepare the new test connection ingestion+UI logic (#10660) * Prepare the new test connection ingestion logic * Update test assert * Update Test Connection for SQA Sources * Correct return type and method doc * Handle decryption * Non SQA Database Sources * Add the run_automation script in ingestion-base * Dashboard Test Connection Changes * Pipeline, Messagin, MlModel & Metadata Sources * ui: test connect flow-1 * Unmask connection parameters before sending to Ariflow * ui: test connect flow-2 * Address review comments and pylint * pytest fix * ui: test connect flow-3 (refactoring and style fix) * ui: test connect flow-4 (fix test connection status logic) * sync local file * ui: test connect flow-5 (fix lowercase issue and styling) * ui: test connect flow-5 (show toast notifications) * test: add unit test * ui: test connect flow-5 (update service page test connection button) * Databrick fix & pytest fix * pylint * Update test * Fix merge * S3 Test connection * add style for mandatory step * sync locales * chore: add service name in workflow request * Unmask using original service connection parameters * Fix test connection unmasking * Wrap inspector function to eliminate error outside test conn * Fix linting * fix:cy test * Fix linting * address comment * refactor and fix connection type casing issue --------- Co-authored-by: ulixius9 <mayursingal9@gmail.com> Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com> Co-authored-by: Nahuel Verdugo Revigliono <nahuel@getcollate.io> Co-authored-by: Mayur Singal <39544459+ulixius9@users.noreply.github.com>
2023-03-28 06:29:13 +02:00
def main():
"""
We will receive in the config env var the Automations Workflow YAML,
for example, for a Test Connection Request:
```json
id: d8afbfd4-f8da-4e0b-8b37-53a26a774d54
name: test-connection-mysql-01
description: mysql test connection workflow
fullyQualifiedName: test-connection-mysql-01
workflowType: TEST_CONNECTION
status: Successful
request:
connection:
config:
type: Mysql
scheme: mysql+pymysql
username: openmetadata_user
password: openmetadata_password
hostPort: 'mysql:3306'
serviceType: Database
connectionType: Mysql
serviceName: mysql_local_01
secretsManagerProvider: noop
openMetadataServerConnection:
clusterName: openmetadata
type: OpenMetadata
hostPort: 'http://openmetadata-server:8585/api'
authProvider: openmetadata
```
"""
config = os.getenv("config")
if not config:
raise RuntimeError(
"Missing environment variable `config` with the Automations Workflow dict."
)
# Default test connection to INFO logs
set_loggers_level(logging.INFO)
[WIP] - Test Connection - Prepare the new test connection ingestion+UI logic (#10660) * Prepare the new test connection ingestion logic * Update test assert * Update Test Connection for SQA Sources * Correct return type and method doc * Handle decryption * Non SQA Database Sources * Add the run_automation script in ingestion-base * Dashboard Test Connection Changes * Pipeline, Messagin, MlModel & Metadata Sources * ui: test connect flow-1 * Unmask connection parameters before sending to Ariflow * ui: test connect flow-2 * Address review comments and pylint * pytest fix * ui: test connect flow-3 (refactoring and style fix) * ui: test connect flow-4 (fix test connection status logic) * sync local file * ui: test connect flow-5 (fix lowercase issue and styling) * ui: test connect flow-5 (show toast notifications) * test: add unit test * ui: test connect flow-5 (update service page test connection button) * Databrick fix & pytest fix * pylint * Update test * Fix merge * S3 Test connection * add style for mandatory step * sync locales * chore: add service name in workflow request * Unmask using original service connection parameters * Fix test connection unmasking * Wrap inspector function to eliminate error outside test conn * Fix linting * fix:cy test * Fix linting * address comment * refactor and fix connection type casing issue --------- Co-authored-by: ulixius9 <mayursingal9@gmail.com> Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com> Co-authored-by: Nahuel Verdugo Revigliono <nahuel@getcollate.io> Co-authored-by: Mayur Singal <39544459+ulixius9@users.noreply.github.com>
2023-03-28 06:29:13 +02:00
automation_workflow_dict = yaml.safe_load(config)
automation_workflow = AutomationWorkflow.model_validate(automation_workflow_dict)
[WIP] - Test Connection - Prepare the new test connection ingestion+UI logic (#10660) * Prepare the new test connection ingestion logic * Update test assert * Update Test Connection for SQA Sources * Correct return type and method doc * Handle decryption * Non SQA Database Sources * Add the run_automation script in ingestion-base * Dashboard Test Connection Changes * Pipeline, Messagin, MlModel & Metadata Sources * ui: test connect flow-1 * Unmask connection parameters before sending to Ariflow * ui: test connect flow-2 * Address review comments and pylint * pytest fix * ui: test connect flow-3 (refactoring and style fix) * ui: test connect flow-4 (fix test connection status logic) * sync local file * ui: test connect flow-5 (fix lowercase issue and styling) * ui: test connect flow-5 (show toast notifications) * test: add unit test * ui: test connect flow-5 (update service page test connection button) * Databrick fix & pytest fix * pylint * Update test * Fix merge * S3 Test connection * add style for mandatory step * sync locales * chore: add service name in workflow request * Unmask using original service connection parameters * Fix test connection unmasking * Wrap inspector function to eliminate error outside test conn * Fix linting * fix:cy test * Fix linting * address comment * refactor and fix connection type casing issue --------- Co-authored-by: ulixius9 <mayursingal9@gmail.com> Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com> Co-authored-by: Nahuel Verdugo Revigliono <nahuel@getcollate.io> Co-authored-by: Mayur Singal <39544459+ulixius9@users.noreply.github.com>
2023-03-28 06:29:13 +02:00
execute(automation_workflow)
if __name__ == "__main__":
main()