Co-authored-by: Akash Verma <akashverma@Akashs-MacBook-Pro-2.local>
This commit is contained in:
Akash Verma 2025-04-03 15:01:45 +05:30 committed by GitHub
parent d2c1705a0b
commit e6a50794c0
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
7 changed files with 201 additions and 0 deletions

View File

@ -16,5 +16,6 @@
{% connectorInfoCard name="NiFi" stage="PROD" href="/connectors/pipeline/nifi" platform="OpenMetadata" / %}
{% connectorInfoCard name="Spline" stage="BETA" href="/connectors/pipeline/spline" platform="OpenMetadata" / %}
{% connectorInfoCard name="Stitch" stage="PROD" href="/connectors/pipeline/stitch" platform="Collate" / %}
{% connectorInfoCard name="Wherescape" stage="BETA" href="/connectors/pipeline/wherescape" platform="Collate" / %}
{% /connectorsListContainer %}

View File

@ -375,6 +375,10 @@ site_menu:
url: /connectors/pipeline/airflow/mwaa
- category: Connectors / Pipeline / Airflow / GCS Composer
url: /connectors/pipeline/airflow/gcs-composer
- category: Connectors / Pipeline / Wherescape
url: /connectors/pipeline/wherescape
- category: Connectors / Pipeline / Wherescape / Run Externally
url: /connectors/pipeline/wherescape/yaml
- category: Connectors / Pipeline / Dagster
url: /connectors/pipeline/dagster
- category: Connectors / Pipeline / Dagster / Run Externally

View File

@ -0,0 +1,74 @@
---
title: Wherescape
slug: /connectors/pipeline/wherescape
---
{% connectorDetailsHeader
name="Wherescape"
stage="BETA"
platform="Collate"
availableFeatures=["Pipelines", "Pipeline Status", "Lineage", "Owners"]
unavailableFeatures=["Tags"]
/ %}
In this section, we provide guides and references to use the Wherescape connector.
Configure and schedule Wherescape metadata workflow from the OpenMetadata UI:
- [Requirements](#requirements)
- [Metadata Ingestion](#metadata-ingestion)
- [Connection Details](#connection-details)
{% partial file="/v1.6/connectors/ingestion-modes-tiles.md" variables={yamlPath: "/connectors/pipeline/wherescape/yaml"} /%}
## Requirements
{% note %}
The Wherescape connector is available only in OpenMetadata version 1.6.8 or later.
{% /note %}
To extract Wherescape metadata, we need the batabase connection details where the metadata is stored.
- `API` Permission ( While Creating the User, from Admin -> User )
- To retrieve lineage data, the user must be granted [Component-level permissions](https://docs.matillion.com/metl/docs/2932106/#component).
- To enable lineage tracking in Matillion, **Matillion Enterprise Mode** is required. For detailed setup instructions and further information, refer to the official documentation: [Matillion Lineage Documentation](https://docs.matillion.com/metl/docs/2881895/).
## Metadata Ingestion
{% partial
file="/v1.6/connectors/metadata-ingestion-ui.md"
variables={
connector: "Wherescape",
selectServicePath: "/images/v1.6/connectors/wherescape/select-service.png",
addNewServicePath: "/images/v1.6/connectors/wherescape/add-new-service.png",
serviceConnectionPath: "/images/v1.6/connectors/wherescape/service-connection.png",
}
/%}
{% stepsContainer %}
{% extraContent parentTagName="stepsContainer" %}
#### Connection Details
- **Connection**: Wherescape metadata database connection.
In terms of `connection` we support the following selections:
- **Microsoft SQL Server**: To connect to the Wherescape metadata database:
- Provide the SQL Server connection credentials including username and password
- Specify the database name where Wherescape metadata is stored
- Enter the host and port for the SQL Server instance
- The connector will establish a connection to this database to extract Wherescape pipeline metadata
{% /extraContent %}
{% partial file="/v1.6/connectors/test-connection.md" /%}
{% partial file="/v1.6/connectors/pipeline/configure-ingestion.md" /%}
{% partial file="/v1.6/connectors/ingestion-schedule-and-deploy.md" /%}
{% /stepsContainer %}
{% partial file="/v1.6/connectors/troubleshooting.md" /%}

View File

@ -0,0 +1,122 @@
---
title: Run the Wherescape Connector Externally
slug: /connectors/pipeline/wherescape/yaml
---
{% connectorDetailsHeader
name="Wherescape"
stage="BETA"
platform="Collate"
availableFeatures=["Pipelines", "Pipeline Status", "Lineage", "Owners"]
unavailableFeatures=["Tags"]
/ %}
In this section, we provide guides and references to use the Wherescape connector.
Configure and schedule Wherescape metadata workflows from the OpenMetadata UI:
- [Requirements](#requirements)
- [Metadata Ingestion](#metadata-ingestion)
{% partial file="/v1.6/connectors/external-ingestion-deployment.md" /%}
## Requirements
{% note %}
The Wherescape connector is available only in OpenMetadata version 1.6.8 or later.
{% /note %}
### Python Requirements
{% partial file="/v1.6/connectors/python-requirements.md" /%}
To run the Wherescape ingestion, you will need to install:
```bash
pip3 install "openmetadata-ingestion[wherescape]"
```
{% /note %}
## Metadata Ingestion
All connectors are defined as JSON Schemas.
[Here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/entity/services/connections/pipeline/wherescapeConnection.json)
you can find the structure to create a connection to Wherescape.
In order to create and run a Metadata Ingestion workflow, we will follow
the steps to create a YAML configuration able to connect to the source,
process the Entities if needed, and reach the OpenMetadata server.
The workflow is modeled around the following
[JSON Schema](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/metadataIngestion/workflow.json)
### 1. Define the YAML Config
This is a sample config for Wherescape:
{% codePreview %}
{% codeInfoContainer %}
#### Source Configuration - Service Connection
{% codeInfo srNumber=1 %}
{% /codeInfo %}
{% codeInfo srNumber=1 %}
**connection**:
In terms of `connection` we support the following selections:
- `Mssql`: Pass the required credentials to reach out this service. We will
create a connection to the pointed database and read Wherescape data from there.
{% /codeInfo %}
{% partial file="/v1.6/connectors/yaml/pipeline/source-config-def.md" /%}
{% partial file="/v1.6/connectors/yaml/ingestion-sink-def.md" /%}
{% partial file="/v1.6/connectors/yaml/workflow-config-def.md" /%}
{% /codeInfoContainer %}
{% codeBlock fileName="filename.yaml" %}
```yaml {% isCodeBlock=true %}
source:
type: wherescape
serviceName: wherescape_source
serviceConnection:
config:
type: Wherescape
```
```yaml {% srNumber=6 %}
# Connection needs to be Mssql
connection:
type: Mssql
username: user
authType:
password: pass
databaseSchema: db
hostPort: localhost:1433
```
{% partial file="/v1.6/connectors/yaml/pipeline/source-config.md" /%}
{% partial file="/v1.6/connectors/yaml/ingestion-sink.md" /%}
{% partial file="/v1.6/connectors/yaml/workflow-config.md" /%}
{% /codeBlock %}
{% /codePreview %}
{% partial file="/v1.6/connectors/yaml/ingestion-cli.md" /%}

Binary file not shown.

After

Width:  |  Height:  |  Size: 88 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 147 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 222 KiB