mirror of
https://github.com/open-metadata/OpenMetadata.git
synced 2025-11-01 11:09:14 +00:00
ws docs (#20606)
Co-authored-by: Akash Verma <akashverma@Akashs-MacBook-Pro-2.local>
This commit is contained in:
parent
d2c1705a0b
commit
e6a50794c0
@ -16,5 +16,6 @@
|
||||
{% connectorInfoCard name="NiFi" stage="PROD" href="/connectors/pipeline/nifi" platform="OpenMetadata" / %}
|
||||
{% connectorInfoCard name="Spline" stage="BETA" href="/connectors/pipeline/spline" platform="OpenMetadata" / %}
|
||||
{% connectorInfoCard name="Stitch" stage="PROD" href="/connectors/pipeline/stitch" platform="Collate" / %}
|
||||
{% connectorInfoCard name="Wherescape" stage="BETA" href="/connectors/pipeline/wherescape" platform="Collate" / %}
|
||||
|
||||
{% /connectorsListContainer %}
|
||||
@ -375,6 +375,10 @@ site_menu:
|
||||
url: /connectors/pipeline/airflow/mwaa
|
||||
- category: Connectors / Pipeline / Airflow / GCS Composer
|
||||
url: /connectors/pipeline/airflow/gcs-composer
|
||||
- category: Connectors / Pipeline / Wherescape
|
||||
url: /connectors/pipeline/wherescape
|
||||
- category: Connectors / Pipeline / Wherescape / Run Externally
|
||||
url: /connectors/pipeline/wherescape/yaml
|
||||
- category: Connectors / Pipeline / Dagster
|
||||
url: /connectors/pipeline/dagster
|
||||
- category: Connectors / Pipeline / Dagster / Run Externally
|
||||
|
||||
@ -0,0 +1,74 @@
|
||||
---
|
||||
title: Wherescape
|
||||
slug: /connectors/pipeline/wherescape
|
||||
---
|
||||
|
||||
{% connectorDetailsHeader
|
||||
name="Wherescape"
|
||||
stage="BETA"
|
||||
platform="Collate"
|
||||
availableFeatures=["Pipelines", "Pipeline Status", "Lineage", "Owners"]
|
||||
unavailableFeatures=["Tags"]
|
||||
/ %}
|
||||
|
||||
In this section, we provide guides and references to use the Wherescape connector.
|
||||
|
||||
Configure and schedule Wherescape metadata workflow from the OpenMetadata UI:
|
||||
|
||||
- [Requirements](#requirements)
|
||||
- [Metadata Ingestion](#metadata-ingestion)
|
||||
- [Connection Details](#connection-details)
|
||||
|
||||
{% partial file="/v1.6/connectors/ingestion-modes-tiles.md" variables={yamlPath: "/connectors/pipeline/wherescape/yaml"} /%}
|
||||
|
||||
## Requirements
|
||||
{% note %}
|
||||
|
||||
The Wherescape connector is available only in OpenMetadata version 1.6.8 or later.
|
||||
|
||||
{% /note %}
|
||||
|
||||
To extract Wherescape metadata, we need the batabase connection details where the metadata is stored.
|
||||
|
||||
- `API` Permission ( While Creating the User, from Admin -> User )
|
||||
- To retrieve lineage data, the user must be granted [Component-level permissions](https://docs.matillion.com/metl/docs/2932106/#component).
|
||||
- To enable lineage tracking in Matillion, **Matillion Enterprise Mode** is required. For detailed setup instructions and further information, refer to the official documentation: [Matillion Lineage Documentation](https://docs.matillion.com/metl/docs/2881895/).
|
||||
|
||||
## Metadata Ingestion
|
||||
|
||||
{% partial
|
||||
file="/v1.6/connectors/metadata-ingestion-ui.md"
|
||||
variables={
|
||||
connector: "Wherescape",
|
||||
selectServicePath: "/images/v1.6/connectors/wherescape/select-service.png",
|
||||
addNewServicePath: "/images/v1.6/connectors/wherescape/add-new-service.png",
|
||||
serviceConnectionPath: "/images/v1.6/connectors/wherescape/service-connection.png",
|
||||
}
|
||||
/%}
|
||||
|
||||
{% stepsContainer %}
|
||||
{% extraContent parentTagName="stepsContainer" %}
|
||||
|
||||
#### Connection Details
|
||||
|
||||
- **Connection**: Wherescape metadata database connection.
|
||||
|
||||
In terms of `connection` we support the following selections:
|
||||
|
||||
- **Microsoft SQL Server**: To connect to the Wherescape metadata database:
|
||||
- Provide the SQL Server connection credentials including username and password
|
||||
- Specify the database name where Wherescape metadata is stored
|
||||
- Enter the host and port for the SQL Server instance
|
||||
- The connector will establish a connection to this database to extract Wherescape pipeline metadata
|
||||
|
||||
{% /extraContent %}
|
||||
|
||||
{% partial file="/v1.6/connectors/test-connection.md" /%}
|
||||
|
||||
{% partial file="/v1.6/connectors/pipeline/configure-ingestion.md" /%}
|
||||
|
||||
{% partial file="/v1.6/connectors/ingestion-schedule-and-deploy.md" /%}
|
||||
|
||||
{% /stepsContainer %}
|
||||
|
||||
{% partial file="/v1.6/connectors/troubleshooting.md" /%}
|
||||
@ -0,0 +1,122 @@
|
||||
---
|
||||
title: Run the Wherescape Connector Externally
|
||||
slug: /connectors/pipeline/wherescape/yaml
|
||||
---
|
||||
|
||||
{% connectorDetailsHeader
|
||||
name="Wherescape"
|
||||
stage="BETA"
|
||||
platform="Collate"
|
||||
availableFeatures=["Pipelines", "Pipeline Status", "Lineage", "Owners"]
|
||||
unavailableFeatures=["Tags"]
|
||||
/ %}
|
||||
|
||||
In this section, we provide guides and references to use the Wherescape connector.
|
||||
|
||||
Configure and schedule Wherescape metadata workflows from the OpenMetadata UI:
|
||||
|
||||
- [Requirements](#requirements)
|
||||
- [Metadata Ingestion](#metadata-ingestion)
|
||||
|
||||
{% partial file="/v1.6/connectors/external-ingestion-deployment.md" /%}
|
||||
|
||||
## Requirements
|
||||
{% note %}
|
||||
|
||||
The Wherescape connector is available only in OpenMetadata version 1.6.8 or later.
|
||||
|
||||
{% /note %}
|
||||
|
||||
### Python Requirements
|
||||
|
||||
{% partial file="/v1.6/connectors/python-requirements.md" /%}
|
||||
|
||||
To run the Wherescape ingestion, you will need to install:
|
||||
|
||||
```bash
|
||||
pip3 install "openmetadata-ingestion[wherescape]"
|
||||
```
|
||||
|
||||
|
||||
{% /note %}
|
||||
|
||||
|
||||
## Metadata Ingestion
|
||||
|
||||
All connectors are defined as JSON Schemas.
|
||||
[Here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/entity/services/connections/pipeline/wherescapeConnection.json)
|
||||
you can find the structure to create a connection to Wherescape.
|
||||
|
||||
In order to create and run a Metadata Ingestion workflow, we will follow
|
||||
the steps to create a YAML configuration able to connect to the source,
|
||||
process the Entities if needed, and reach the OpenMetadata server.
|
||||
|
||||
The workflow is modeled around the following
|
||||
[JSON Schema](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/metadataIngestion/workflow.json)
|
||||
|
||||
### 1. Define the YAML Config
|
||||
|
||||
This is a sample config for Wherescape:
|
||||
|
||||
{% codePreview %}
|
||||
|
||||
{% codeInfoContainer %}
|
||||
|
||||
#### Source Configuration - Service Connection
|
||||
|
||||
{% codeInfo srNumber=1 %}
|
||||
|
||||
{% /codeInfo %}
|
||||
|
||||
{% codeInfo srNumber=1 %}
|
||||
|
||||
**connection**:
|
||||
|
||||
In terms of `connection` we support the following selections:
|
||||
|
||||
- `Mssql`: Pass the required credentials to reach out this service. We will
|
||||
create a connection to the pointed database and read Wherescape data from there.
|
||||
|
||||
{% /codeInfo %}
|
||||
|
||||
{% partial file="/v1.6/connectors/yaml/pipeline/source-config-def.md" /%}
|
||||
|
||||
{% partial file="/v1.6/connectors/yaml/ingestion-sink-def.md" /%}
|
||||
|
||||
{% partial file="/v1.6/connectors/yaml/workflow-config-def.md" /%}
|
||||
|
||||
{% /codeInfoContainer %}
|
||||
|
||||
{% codeBlock fileName="filename.yaml" %}
|
||||
|
||||
```yaml {% isCodeBlock=true %}
|
||||
source:
|
||||
type: wherescape
|
||||
serviceName: wherescape_source
|
||||
serviceConnection:
|
||||
config:
|
||||
type: Wherescape
|
||||
```
|
||||
```yaml {% srNumber=6 %}
|
||||
# Connection needs to be Mssql
|
||||
connection:
|
||||
type: Mssql
|
||||
username: user
|
||||
authType:
|
||||
password: pass
|
||||
databaseSchema: db
|
||||
hostPort: localhost:1433
|
||||
```
|
||||
|
||||
{% partial file="/v1.6/connectors/yaml/pipeline/source-config.md" /%}
|
||||
|
||||
{% partial file="/v1.6/connectors/yaml/ingestion-sink.md" /%}
|
||||
|
||||
{% partial file="/v1.6/connectors/yaml/workflow-config.md" /%}
|
||||
|
||||
{% /codeBlock %}
|
||||
|
||||
{% /codePreview %}
|
||||
|
||||
{% partial file="/v1.6/connectors/yaml/ingestion-cli.md" /%}
|
||||
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 88 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 147 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 222 KiB |
Loading…
x
Reference in New Issue
Block a user