mirror of
https://github.com/open-metadata/OpenMetadata.git
synced 2025-11-08 23:14:00 +00:00
ws docs (#20606)
Co-authored-by: Akash Verma <akashverma@Akashs-MacBook-Pro-2.local>
This commit is contained in:
parent
d2c1705a0b
commit
e6a50794c0
@ -16,5 +16,6 @@
|
|||||||
{% connectorInfoCard name="NiFi" stage="PROD" href="/connectors/pipeline/nifi" platform="OpenMetadata" / %}
|
{% connectorInfoCard name="NiFi" stage="PROD" href="/connectors/pipeline/nifi" platform="OpenMetadata" / %}
|
||||||
{% connectorInfoCard name="Spline" stage="BETA" href="/connectors/pipeline/spline" platform="OpenMetadata" / %}
|
{% connectorInfoCard name="Spline" stage="BETA" href="/connectors/pipeline/spline" platform="OpenMetadata" / %}
|
||||||
{% connectorInfoCard name="Stitch" stage="PROD" href="/connectors/pipeline/stitch" platform="Collate" / %}
|
{% connectorInfoCard name="Stitch" stage="PROD" href="/connectors/pipeline/stitch" platform="Collate" / %}
|
||||||
|
{% connectorInfoCard name="Wherescape" stage="BETA" href="/connectors/pipeline/wherescape" platform="Collate" / %}
|
||||||
|
|
||||||
{% /connectorsListContainer %}
|
{% /connectorsListContainer %}
|
||||||
@ -375,6 +375,10 @@ site_menu:
|
|||||||
url: /connectors/pipeline/airflow/mwaa
|
url: /connectors/pipeline/airflow/mwaa
|
||||||
- category: Connectors / Pipeline / Airflow / GCS Composer
|
- category: Connectors / Pipeline / Airflow / GCS Composer
|
||||||
url: /connectors/pipeline/airflow/gcs-composer
|
url: /connectors/pipeline/airflow/gcs-composer
|
||||||
|
- category: Connectors / Pipeline / Wherescape
|
||||||
|
url: /connectors/pipeline/wherescape
|
||||||
|
- category: Connectors / Pipeline / Wherescape / Run Externally
|
||||||
|
url: /connectors/pipeline/wherescape/yaml
|
||||||
- category: Connectors / Pipeline / Dagster
|
- category: Connectors / Pipeline / Dagster
|
||||||
url: /connectors/pipeline/dagster
|
url: /connectors/pipeline/dagster
|
||||||
- category: Connectors / Pipeline / Dagster / Run Externally
|
- category: Connectors / Pipeline / Dagster / Run Externally
|
||||||
|
|||||||
@ -0,0 +1,74 @@
|
|||||||
|
---
|
||||||
|
title: Wherescape
|
||||||
|
slug: /connectors/pipeline/wherescape
|
||||||
|
---
|
||||||
|
|
||||||
|
{% connectorDetailsHeader
|
||||||
|
name="Wherescape"
|
||||||
|
stage="BETA"
|
||||||
|
platform="Collate"
|
||||||
|
availableFeatures=["Pipelines", "Pipeline Status", "Lineage", "Owners"]
|
||||||
|
unavailableFeatures=["Tags"]
|
||||||
|
/ %}
|
||||||
|
|
||||||
|
In this section, we provide guides and references to use the Wherescape connector.
|
||||||
|
|
||||||
|
Configure and schedule Wherescape metadata workflow from the OpenMetadata UI:
|
||||||
|
|
||||||
|
- [Requirements](#requirements)
|
||||||
|
- [Metadata Ingestion](#metadata-ingestion)
|
||||||
|
- [Connection Details](#connection-details)
|
||||||
|
|
||||||
|
{% partial file="/v1.6/connectors/ingestion-modes-tiles.md" variables={yamlPath: "/connectors/pipeline/wherescape/yaml"} /%}
|
||||||
|
|
||||||
|
## Requirements
|
||||||
|
{% note %}
|
||||||
|
|
||||||
|
The Wherescape connector is available only in OpenMetadata version 1.6.8 or later.
|
||||||
|
|
||||||
|
{% /note %}
|
||||||
|
|
||||||
|
To extract Wherescape metadata, we need the batabase connection details where the metadata is stored.
|
||||||
|
|
||||||
|
- `API` Permission ( While Creating the User, from Admin -> User )
|
||||||
|
- To retrieve lineage data, the user must be granted [Component-level permissions](https://docs.matillion.com/metl/docs/2932106/#component).
|
||||||
|
- To enable lineage tracking in Matillion, **Matillion Enterprise Mode** is required. For detailed setup instructions and further information, refer to the official documentation: [Matillion Lineage Documentation](https://docs.matillion.com/metl/docs/2881895/).
|
||||||
|
|
||||||
|
## Metadata Ingestion
|
||||||
|
|
||||||
|
{% partial
|
||||||
|
file="/v1.6/connectors/metadata-ingestion-ui.md"
|
||||||
|
variables={
|
||||||
|
connector: "Wherescape",
|
||||||
|
selectServicePath: "/images/v1.6/connectors/wherescape/select-service.png",
|
||||||
|
addNewServicePath: "/images/v1.6/connectors/wherescape/add-new-service.png",
|
||||||
|
serviceConnectionPath: "/images/v1.6/connectors/wherescape/service-connection.png",
|
||||||
|
}
|
||||||
|
/%}
|
||||||
|
|
||||||
|
{% stepsContainer %}
|
||||||
|
{% extraContent parentTagName="stepsContainer" %}
|
||||||
|
|
||||||
|
#### Connection Details
|
||||||
|
|
||||||
|
- **Connection**: Wherescape metadata database connection.
|
||||||
|
|
||||||
|
In terms of `connection` we support the following selections:
|
||||||
|
|
||||||
|
- **Microsoft SQL Server**: To connect to the Wherescape metadata database:
|
||||||
|
- Provide the SQL Server connection credentials including username and password
|
||||||
|
- Specify the database name where Wherescape metadata is stored
|
||||||
|
- Enter the host and port for the SQL Server instance
|
||||||
|
- The connector will establish a connection to this database to extract Wherescape pipeline metadata
|
||||||
|
|
||||||
|
{% /extraContent %}
|
||||||
|
|
||||||
|
{% partial file="/v1.6/connectors/test-connection.md" /%}
|
||||||
|
|
||||||
|
{% partial file="/v1.6/connectors/pipeline/configure-ingestion.md" /%}
|
||||||
|
|
||||||
|
{% partial file="/v1.6/connectors/ingestion-schedule-and-deploy.md" /%}
|
||||||
|
|
||||||
|
{% /stepsContainer %}
|
||||||
|
|
||||||
|
{% partial file="/v1.6/connectors/troubleshooting.md" /%}
|
||||||
@ -0,0 +1,122 @@
|
|||||||
|
---
|
||||||
|
title: Run the Wherescape Connector Externally
|
||||||
|
slug: /connectors/pipeline/wherescape/yaml
|
||||||
|
---
|
||||||
|
|
||||||
|
{% connectorDetailsHeader
|
||||||
|
name="Wherescape"
|
||||||
|
stage="BETA"
|
||||||
|
platform="Collate"
|
||||||
|
availableFeatures=["Pipelines", "Pipeline Status", "Lineage", "Owners"]
|
||||||
|
unavailableFeatures=["Tags"]
|
||||||
|
/ %}
|
||||||
|
|
||||||
|
In this section, we provide guides and references to use the Wherescape connector.
|
||||||
|
|
||||||
|
Configure and schedule Wherescape metadata workflows from the OpenMetadata UI:
|
||||||
|
|
||||||
|
- [Requirements](#requirements)
|
||||||
|
- [Metadata Ingestion](#metadata-ingestion)
|
||||||
|
|
||||||
|
{% partial file="/v1.6/connectors/external-ingestion-deployment.md" /%}
|
||||||
|
|
||||||
|
## Requirements
|
||||||
|
{% note %}
|
||||||
|
|
||||||
|
The Wherescape connector is available only in OpenMetadata version 1.6.8 or later.
|
||||||
|
|
||||||
|
{% /note %}
|
||||||
|
|
||||||
|
### Python Requirements
|
||||||
|
|
||||||
|
{% partial file="/v1.6/connectors/python-requirements.md" /%}
|
||||||
|
|
||||||
|
To run the Wherescape ingestion, you will need to install:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
pip3 install "openmetadata-ingestion[wherescape]"
|
||||||
|
```
|
||||||
|
|
||||||
|
|
||||||
|
{% /note %}
|
||||||
|
|
||||||
|
|
||||||
|
## Metadata Ingestion
|
||||||
|
|
||||||
|
All connectors are defined as JSON Schemas.
|
||||||
|
[Here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/entity/services/connections/pipeline/wherescapeConnection.json)
|
||||||
|
you can find the structure to create a connection to Wherescape.
|
||||||
|
|
||||||
|
In order to create and run a Metadata Ingestion workflow, we will follow
|
||||||
|
the steps to create a YAML configuration able to connect to the source,
|
||||||
|
process the Entities if needed, and reach the OpenMetadata server.
|
||||||
|
|
||||||
|
The workflow is modeled around the following
|
||||||
|
[JSON Schema](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/metadataIngestion/workflow.json)
|
||||||
|
|
||||||
|
### 1. Define the YAML Config
|
||||||
|
|
||||||
|
This is a sample config for Wherescape:
|
||||||
|
|
||||||
|
{% codePreview %}
|
||||||
|
|
||||||
|
{% codeInfoContainer %}
|
||||||
|
|
||||||
|
#### Source Configuration - Service Connection
|
||||||
|
|
||||||
|
{% codeInfo srNumber=1 %}
|
||||||
|
|
||||||
|
{% /codeInfo %}
|
||||||
|
|
||||||
|
{% codeInfo srNumber=1 %}
|
||||||
|
|
||||||
|
**connection**:
|
||||||
|
|
||||||
|
In terms of `connection` we support the following selections:
|
||||||
|
|
||||||
|
- `Mssql`: Pass the required credentials to reach out this service. We will
|
||||||
|
create a connection to the pointed database and read Wherescape data from there.
|
||||||
|
|
||||||
|
{% /codeInfo %}
|
||||||
|
|
||||||
|
{% partial file="/v1.6/connectors/yaml/pipeline/source-config-def.md" /%}
|
||||||
|
|
||||||
|
{% partial file="/v1.6/connectors/yaml/ingestion-sink-def.md" /%}
|
||||||
|
|
||||||
|
{% partial file="/v1.6/connectors/yaml/workflow-config-def.md" /%}
|
||||||
|
|
||||||
|
{% /codeInfoContainer %}
|
||||||
|
|
||||||
|
{% codeBlock fileName="filename.yaml" %}
|
||||||
|
|
||||||
|
```yaml {% isCodeBlock=true %}
|
||||||
|
source:
|
||||||
|
type: wherescape
|
||||||
|
serviceName: wherescape_source
|
||||||
|
serviceConnection:
|
||||||
|
config:
|
||||||
|
type: Wherescape
|
||||||
|
```
|
||||||
|
```yaml {% srNumber=6 %}
|
||||||
|
# Connection needs to be Mssql
|
||||||
|
connection:
|
||||||
|
type: Mssql
|
||||||
|
username: user
|
||||||
|
authType:
|
||||||
|
password: pass
|
||||||
|
databaseSchema: db
|
||||||
|
hostPort: localhost:1433
|
||||||
|
```
|
||||||
|
|
||||||
|
{% partial file="/v1.6/connectors/yaml/pipeline/source-config.md" /%}
|
||||||
|
|
||||||
|
{% partial file="/v1.6/connectors/yaml/ingestion-sink.md" /%}
|
||||||
|
|
||||||
|
{% partial file="/v1.6/connectors/yaml/workflow-config.md" /%}
|
||||||
|
|
||||||
|
{% /codeBlock %}
|
||||||
|
|
||||||
|
{% /codePreview %}
|
||||||
|
|
||||||
|
{% partial file="/v1.6/connectors/yaml/ingestion-cli.md" /%}
|
||||||
|
|
||||||
Binary file not shown.
|
After Width: | Height: | Size: 88 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 147 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 222 KiB |
Loading…
x
Reference in New Issue
Block a user