yujunjun/OpenMetadata
1
0
Fork 0
mirror of https://github.com/open-metadata/OpenMetadata.git synced 2025-11-02 03:29:03 +00:00
Code Issues Packages Projects Releases Wiki Activity

Add Azure Datalake to the list (#9487)

Browse Source 
* Add Azure Datalake to the list

* Put Yaml configs under the relevant sections
This commit is contained in:
Ayush Shah 2022-12-22 15:45:57 +05:30 committed by GitHub
parent 03f2b564cc
commit b3d7719289
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
2 changed files with 10 additions and 7 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

16
openmetadata-docs/content/connectors/database/datalake/cli.md
View File

@ -88,6 +88,9 @@ In order to create and run a Metadata Ingestion workflow, we will follow the ste
The workflow is modeled around the following JSON Schema.
## 1. Define the YAML Config
#### Source Configuration - Source Config using AWS S3
This is a sample config for Datalake using AWS S3:
```yaml
@ -121,8 +124,6 @@ workflowConfig:
```
#### Source Configuration - Source Config using AWS S3
The `sourceConfig` is defined [here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/metadataIngestion/databaseServiceMetadataPipeline.json).
* **awsAccessKeyId**: Enter your secure access key ID for your DynamoDB connection. The specified key ID should be authorized to read all databases you want to include in the metadata ingestion workflow.
@ -130,6 +131,9 @@ The `sourceConfig` is defined [here](https://github.com/open-metadata/OpenMetada
* **awsRegion**: Specify the region in which your DynamoDB is located. This setting is required even if you have configured a local AWS profile.
* **schemaFilterPattern** and **tableFilternPattern**: Note that the `schemaFilterPattern` and `tableFilterPattern` both support regex as `include` or `exclude`. E.g.,
#### Source Configuration - Service Connection using GCS
This is a sample config for Datalake using GCS:
```yaml
@ -169,9 +173,6 @@ workflowConfig:
authProvider: <OpenMetadata auth provider>
```
#### Source Configuration - Service Connection using GCS
The `sourceConfig` is defined [here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/metadataIngestion/databaseServiceMetadataPipeline.json).
* **type**: Credentials type, e.g. `service_account`.
@ -187,6 +188,9 @@ The `sourceConfig` is defined [here](https://github.com/open-metadata/OpenMetada
* **bucketName**: name of the bucket in GCS
* **Prefix**: prefix in gcs bucket
#### Source Configuration - Service Connection using Azure
This is a sample config for Datalake using Azure:
```yaml
@ -219,8 +223,6 @@ workflowConfig:
authProvider: <OpenMetadata auth provider>
```
#### Source Configuration - Service Connection using Azure
The `sourceConfig` is defined [here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/security/credentials/azureCredentials.json).
- **Client ID** : Client ID of the data storage account

1
openmetadata-docs/content/connectors/index.md
View File

@ -21,6 +21,7 @@ OpenMetadata can extract metadata from the following list of 55 connectors:
- Databricks Metadata
- Databricks Usage
- [Data lake](/connectors/database/datalake)
- Azure Data Lake
- S3 Data Lake
- Google Cloud Service Data Lake
- [DB2](/connectors/database/db2)