mirror of
https://github.com/open-metadata/OpenMetadata.git
synced 2025-11-28 16:45:23 +00:00
Fix Docs - add iam based rds doc (#12210)
* Fix Docs * Fix Yaml * Update ingestion/Dockerfile.ci * Add 1.1.0 changes for impala --------- Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
This commit is contained in:
Ayush Shah
GitHub
parent
f74004d9c8
commit
4aab0fd6fb
@ -76,6 +76,10 @@ source:
|
||||
authOptions: <auth options>
|
||||
authMechanism: PLAIN # NOSASL, PLAIN, GSSAPI, LDAP, JWT
|
||||
hostPort: <impala connection host & port>
|
||||
# kerberosServiceName: KerberosServiceName
|
||||
# databaseSchema: Database Schema of the data source
|
||||
# databaseName: Optional name to give to the database in OpenMetadata.
|
||||
# useSSL: true / false
|
||||
sourceConfig:
|
||||
config:
|
||||
type: DatabaseMetadata
|
||||
@ -327,6 +331,10 @@ source:
|
||||
authOptions: <auth options>
|
||||
authMechanism: PLAIN # NOSASL, PLAIN, GSSAPI, LDAP, JWT
|
||||
hostPort: <impala connection host & port>
|
||||
# kerberosServiceName: KerberosServiceName
|
||||
# databaseSchema: Database Schema of the data source
|
||||
# databaseName: Optional name to give to the database in OpenMetadata.
|
||||
# useSSL: true / false
|
||||
sourceConfig:
|
||||
config:
|
||||
type: Profiler
|
||||
|
||||
@ -138,13 +138,13 @@ For a simple, local installation using our docker containers, this looks like:
|
||||
|
||||
{% codeInfo srNumber=5 %}
|
||||
|
||||
**Connection Options (Optional)**: Enter the details for any additional connection options that can be sent to Athena during the connection. These details must be added as Key-Value pairs.
|
||||
**Connection Options (Optional)**: Enter the details for any additional connection options that can be sent to Mysql during the connection. These details must be added as Key-Value pairs.
|
||||
|
||||
{% /codeInfo %}
|
||||
|
||||
{% codeInfo srNumber=6 %}
|
||||
|
||||
**Connection Arguments (Optional)**: Enter the details for any additional connection arguments such as security or protocol configs that can be sent to Athena during the connection. These details must be added as Key-Value pairs.
|
||||
**Connection Arguments (Optional)**: Enter the details for any additional connection arguments such as security or protocol configs that can be sent to Mysql during the connection. These details must be added as Key-Value pairs.
|
||||
|
||||
- In case you are using Single-Sign-On (SSO) for authentication, add the `authenticator` details in the Connection Arguments as a Key-Value pair as follows: `"authenticator" : "sso_login_url"`
|
||||
|
||||
|
||||
@ -83,17 +83,89 @@ This is a sample config for MySQL:
|
||||
|
||||
{% codeInfo srNumber=2 %}
|
||||
|
||||
**password**: Password to connect to MySQL.
|
||||
**authType**: Choose from basic auth and IAM based auth.
|
||||
#### Basic Auth
|
||||
|
||||
**password**: Password comes under Basic Auth type.
|
||||
|
||||
{% /codeInfo %}
|
||||
|
||||
{% codeInfo srNumber=3 %}
|
||||
#### IAM BASED Auth
|
||||
|
||||
- **awsAccessKeyId** & **awsSecretAccessKey**: When you interact with AWS, you specify your AWS security credentials to verify who you are and whether you have
|
||||
permission to access the resources that you are requesting. AWS uses the security credentials to authenticate and
|
||||
authorize your requests ([docs](https://docs.aws.amazon.com/IAM/latest/UserGuide/security-creds.html)).
|
||||
|
||||
Access keys consist of two parts: An **access key ID** (for example, `AKIAIOSFODNN7EXAMPLE`), and a **secret access key** (for example, `wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY`).
|
||||
|
||||
You must use both the access key ID and secret access key together to authenticate your requests.
|
||||
|
||||
You can find further information on how to manage your access keys [here](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html).
|
||||
|
||||
**awsSessionToken**: If you are using temporary credentials to access your services, you will need to inform the AWS Access Key ID
|
||||
and AWS Secrets Access Key. Also, these will include an AWS Session Token.
|
||||
|
||||
|
||||
**awsRegion**: Each AWS Region is a separate geographic area in which AWS clusters data centers ([docs](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.RegionsAndAvailabilityZones.html)).
|
||||
|
||||
As AWS can have instances in multiple regions, we need to know the region the service you want reach belongs to.
|
||||
|
||||
Note that the AWS Region is the only required parameter when configuring a connection. When connecting to the
|
||||
services programmatically, there are different ways in which we can extract and use the rest of AWS configurations.
|
||||
|
||||
You can find further information about configuring your credentials [here](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html#configuring-credentials).
|
||||
|
||||
|
||||
**endPointURL**: To connect programmatically to an AWS service, you use an endpoint. An *endpoint* is the URL of the
|
||||
entry point for an AWS web service. The AWS SDKs and the AWS Command Line Interface (AWS CLI) automatically use the
|
||||
default endpoint for each service in an AWS Region. But you can specify an alternate endpoint for your API requests.
|
||||
|
||||
Find more information on [AWS service endpoints](https://docs.aws.amazon.com/general/latest/gr/rande.html).
|
||||
|
||||
|
||||
**profileName**: A named profile is a collection of settings and credentials that you can apply to a AWS CLI command.
|
||||
When you specify a profile to run a command, the settings and credentials are used to run that command.
|
||||
Multiple named profiles can be stored in the config and credentials files.
|
||||
|
||||
You can inform this field if you'd like to use a profile other than `default`.
|
||||
|
||||
Find here more information about [Named profiles for the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-profiles.html).
|
||||
|
||||
|
||||
**assumeRoleArn**: Typically, you use `AssumeRole` within your account or for cross-account access. In this field you'll set the
|
||||
`ARN` (Amazon Resource Name) of the policy of the other account.
|
||||
|
||||
A user who wants to access a role in a different account must also have permissions that are delegated from the account
|
||||
administrator. The administrator must attach a policy that allows the user to call `AssumeRole` for the `ARN` of the role in the other account.
|
||||
|
||||
This is a required field if you'd like to `AssumeRole`.
|
||||
|
||||
Find more information on [AssumeRole](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html).
|
||||
|
||||
**assumeRoleSessionName**: An identifier for the assumed role session. Use the role session name to uniquely identify a session when the same role
|
||||
is assumed by different principals or for different reasons.
|
||||
|
||||
By default, we'll use the name `OpenMetadataSession`.
|
||||
|
||||
Find more information about the [Role Session Name](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html#:~:text=An%20identifier%20for%20the%20assumed%20role%20session.).
|
||||
|
||||
|
||||
**assumeRoleSourceIdentity**: The source identity specified by the principal that is calling the `AssumeRole` operation. You can use source identity
|
||||
information in AWS CloudTrail logs to determine who took actions with a role.
|
||||
|
||||
Find more information about [Source Identity](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html#:~:text=Required%3A%20No-,SourceIdentity,-The%20source%20identity).
|
||||
|
||||
{% /codeInfo %}
|
||||
|
||||
|
||||
{% codeInfo srNumber=4 %}
|
||||
|
||||
**Host and Port**: Enter the fully qualified hostname and port number for your MySQL deployment in the Host and Port field.
|
||||
|
||||
{% /codeInfo %}
|
||||
|
||||
{% codeInfo srNumber=4 %}
|
||||
{% codeInfo srNumber=5 %}
|
||||
|
||||
**databaseSchema**: databaseSchema of the data source. This is optional parameter, if you would like to restrict the metadata reading to a single databaseSchema. When left blank, OpenMetadata Ingestion attempts to scan all the databaseSchema.
|
||||
|
||||
@ -101,7 +173,7 @@ This is a sample config for MySQL:
|
||||
|
||||
#### Source Configuration - Source Config
|
||||
|
||||
{% codeInfo srNumber=7 %}
|
||||
{% codeInfo srNumber=8 %}
|
||||
|
||||
The `sourceConfig` is defined [here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/metadataIngestion/databaseServiceMetadataPipeline.json):
|
||||
|
||||
@ -118,7 +190,7 @@ The `sourceConfig` is defined [here](https://github.com/open-metadata/OpenMetada
|
||||
|
||||
#### Sink Configuration
|
||||
|
||||
{% codeInfo srNumber=8 %}
|
||||
{% codeInfo srNumber=9 %}
|
||||
|
||||
To send the metadata to OpenMetadata, it needs to be specified as `type: metadata-rest`.
|
||||
|
||||
@ -126,7 +198,7 @@ To send the metadata to OpenMetadata, it needs to be specified as `type: metadat
|
||||
|
||||
#### Workflow Configuration
|
||||
|
||||
{% codeInfo srNumber=9 %}
|
||||
{% codeInfo srNumber=10 %}
|
||||
|
||||
The main property here is the `openMetadataServerConfig`, where you can define the host and security provider of your OpenMetadata installation.
|
||||
|
||||
@ -136,13 +208,13 @@ For a simple, local installation using our docker containers, this looks like:
|
||||
|
||||
#### Advanced Configuration
|
||||
|
||||
{% codeInfo srNumber=5 %}
|
||||
{% codeInfo srNumber=6 %}
|
||||
|
||||
**Connection Options (Optional)**: Enter the details for any additional connection options that can be sent to Athena during the connection. These details must be added as Key-Value pairs.
|
||||
|
||||
{% /codeInfo %}
|
||||
|
||||
{% codeInfo srNumber=6 %}
|
||||
{% codeInfo srNumber=7 %}
|
||||
|
||||
**Connection Arguments (Optional)**: Enter the details for any additional connection arguments such as security or protocol configs that can be sent to Athena during the connection. These details must be added as Key-Value pairs.
|
||||
|
||||
@ -166,23 +238,31 @@ source:
|
||||
username: <username>
|
||||
```
|
||||
```yaml {% srNumber=2 %}
|
||||
password: <password>
|
||||
authType:
|
||||
password: <password>
|
||||
```
|
||||
```yaml {% srNumber=3 %}
|
||||
hostPort: <hostPort>
|
||||
authType:
|
||||
awsConfig:
|
||||
awsAccessKeyId: access key id
|
||||
awsSecretAccessKey: access secret key
|
||||
awsRegion: aws region name
|
||||
```
|
||||
```yaml {% srNumber=4 %}
|
||||
databaseSchema: schema
|
||||
hostPort: <hostPort>
|
||||
```
|
||||
```yaml {% srNumber=5 %}
|
||||
databaseSchema: schema
|
||||
```
|
||||
```yaml {% srNumber=6 %}
|
||||
# connectionOptions:
|
||||
# key: value
|
||||
```
|
||||
```yaml {% srNumber=6 %}
|
||||
```yaml {% srNumber=7 %}
|
||||
# connectionArguments:
|
||||
# key: value
|
||||
```
|
||||
```yaml {% srNumber=7 %}
|
||||
```yaml {% srNumber=8 %}
|
||||
sourceConfig:
|
||||
config:
|
||||
type: DatabaseMetadata
|
||||
@ -213,13 +293,13 @@ source:
|
||||
# - table4
|
||||
```
|
||||
|
||||
```yaml {% srNumber=8 %}
|
||||
```yaml {% srNumber=9 %}
|
||||
sink:
|
||||
type: metadata-rest
|
||||
config: {}
|
||||
```
|
||||
|
||||
```yaml {% srNumber=9 %}
|
||||
```yaml {% srNumber=10 %}
|
||||
workflowConfig:
|
||||
# loggerLevel: DEBUG # DEBUG, INFO, WARN or ERROR
|
||||
openMetadataServerConfig:
|
||||
|
||||
@ -44,12 +44,12 @@ the following docs to connect using Airflow SDK or with the CLI.
|
||||
{% tile
|
||||
title="Ingest with Airflow"
|
||||
description="Configure the ingestion using Airflow SDK"
|
||||
link="/connectors/database/athena/airflow"
|
||||
link="/connectors/database/mysql/airflow"
|
||||
/ %}
|
||||
{% tile
|
||||
title="Ingest with the CLI"
|
||||
description="Run a one-time ingestion using the metadata CLI"
|
||||
link="/connectors/database/athena/cli"
|
||||
link="/connectors/database/mysql/cli"
|
||||
/ %}
|
||||
|
||||
{% /tilesContainer %}
|
||||
@ -201,7 +201,69 @@ desired.
|
||||
#### Connection Options
|
||||
|
||||
- **Username**: Specify the User to connect to MySQL. It should have enough privileges to read all the metadata.
|
||||
- **Password**: Password to connect to MySQL.
|
||||
- **Auth Type**: Basic Auth or IAM based auth to connect to instances / cloud rds.
|
||||
- **Basic Auth**:
|
||||
- **Password**: Password to connect to MySQL.
|
||||
- **IAM Based Auth**:
|
||||
- **AWS Access Key ID** & **AWS Secret Access Key**: When you interact with AWS, you specify your AWS security credentials to verify who you are and whether you have
|
||||
permission to access the resources that you are requesting. AWS uses the security credentials to authenticate and
|
||||
authorize your requests ([docs](https://docs.aws.amazon.com/IAM/latest/UserGuide/security-creds.html)).
|
||||
|
||||
Access keys consist of two parts: An **access key ID** (for example, `AKIAIOSFODNN7EXAMPLE`), and a **secret access key** (for example, `wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY`).
|
||||
|
||||
You must use both the access key ID and secret access key together to authenticate your requests.
|
||||
|
||||
You can find further information on how to manage your access keys [here](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html).
|
||||
|
||||
- **AWS Region**: Each AWS Region is a separate geographic area in which AWS clusters data centers ([docs](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.RegionsAndAvailabilityZones.html)).
|
||||
|
||||
As AWS can have instances in multiple regions, we need to know the region the service you want reach belongs to.
|
||||
|
||||
Note that the AWS Region is the only required parameter when configuring a connection. When connecting to the
|
||||
services programmatically, there are different ways in which we can extract and use the rest of AWS configurations.
|
||||
|
||||
You can find further information about configuring your credentials [here](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html#configuring-credentials).
|
||||
|
||||
- **AWS Session Token (optional)**: If you are using temporary credentials to access your services, you will need to inform the AWS Access Key ID
|
||||
and AWS Secrets Access Key. Also, these will include an AWS Session Token.
|
||||
|
||||
You can find more information on [Using temporary credentials with AWS resources](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_use-resources.html).
|
||||
|
||||
- **Endpoint URL (optional)**: To connect programmatically to an AWS service, you use an endpoint. An *endpoint* is the URL of the
|
||||
entry point for an AWS web service. The AWS SDKs and the AWS Command Line Interface (AWS CLI) automatically use the
|
||||
default endpoint for each service in an AWS Region. But you can specify an alternate endpoint for your API requests.
|
||||
|
||||
Find more information on [AWS service endpoints](https://docs.aws.amazon.com/general/latest/gr/rande.html).
|
||||
|
||||
- **Profile Name**: A named profile is a collection of settings and credentials that you can apply to a AWS CLI command.
|
||||
When you specify a profile to run a command, the settings and credentials are used to run that command.
|
||||
Multiple named profiles can be stored in the config and credentials files.
|
||||
|
||||
You can inform this field if you'd like to use a profile other than `default`.
|
||||
|
||||
Find here more information about [Named profiles for the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-profiles.html).
|
||||
|
||||
- **Assume Role Arn**: Typically, you use `AssumeRole` within your account or for cross-account access. In this field you'll set the
|
||||
`ARN` (Amazon Resource Name) of the policy of the other account.
|
||||
|
||||
A user who wants to access a role in a different account must also have permissions that are delegated from the account
|
||||
administrator. The administrator must attach a policy that allows the user to call `AssumeRole` for the `ARN` of the role in the other account.
|
||||
|
||||
This is a required field if you'd like to `AssumeRole`.
|
||||
|
||||
Find more information on [AssumeRole](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html).
|
||||
|
||||
- **Assume Role Session Name**: An identifier for the assumed role session. Use the role session name to uniquely identify a session when the same role
|
||||
is assumed by different principals or for different reasons.
|
||||
|
||||
By default, we'll use the name `OpenMetadataSession`.
|
||||
|
||||
Find more information about the [Role Session Name](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html#:~:text=An%20identifier%20for%20the%20assumed%20role%20session.).
|
||||
|
||||
- **Assume Role Source Identity**: The source identity specified by the principal that is calling the `AssumeRole` operation. You can use source identity
|
||||
information in AWS CloudTrail logs to determine who took actions with a role.
|
||||
|
||||
Find more information about [Source Identity](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html#:~:text=Required%3A%20No-,SourceIdentity,-The%20source%20identity).
|
||||
- **Host and Port**: Enter the fully qualified hostname and port number for your MySQL deployment in the Host and Port field.
|
||||
- **databaseName**: Optional name to give to the database in OpenMetadata. If left blank, we will use default as the database name.
|
||||
- **databaseSchema**: databaseSchema of the data source. This is optional parameter, if you would like to restrict the metadata reading to a single databaseSchema. When left blank, OpenMetadata Ingestion attempts to scan all the databaseSchema.
|
||||
|
||||
@ -103,26 +103,95 @@ This is a sample config for Postgres:
|
||||
**username**: Specify the User to connect to Postgres. It should have enough privileges to read all the metadata.
|
||||
|
||||
{% /codeInfo %}
|
||||
|
||||
{% codeInfo srNumber=2 %}
|
||||
|
||||
**password**: Password to connect to Postgres.
|
||||
**authType**: Choose from basic auth and IAM based auth.
|
||||
#### Basic Auth
|
||||
|
||||
**password**: Password comes under Basic Auth type.
|
||||
|
||||
{% /codeInfo %}
|
||||
|
||||
{% codeInfo srNumber=3 %}
|
||||
#### IAM BASED Auth
|
||||
|
||||
- **awsAccessKeyId** & **awsSecretAccessKey**: When you interact with AWS, you specify your AWS security credentials to verify who you are and whether you have
|
||||
permission to access the resources that you are requesting. AWS uses the security credentials to authenticate and
|
||||
authorize your requests ([docs](https://docs.aws.amazon.com/IAM/latest/UserGuide/security-creds.html)).
|
||||
|
||||
Access keys consist of two parts: An **access key ID** (for example, `AKIAIOSFODNN7EXAMPLE`), and a **secret access key** (for example, `wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY`).
|
||||
|
||||
You must use both the access key ID and secret access key together to authenticate your requests.
|
||||
|
||||
You can find further information on how to manage your access keys [here](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html).
|
||||
|
||||
**awsSessionToken**: If you are using temporary credentials to access your services, you will need to inform the AWS Access Key ID
|
||||
and AWS Secrets Access Key. Also, these will include an AWS Session Token.
|
||||
|
||||
|
||||
**awsRegion**: Each AWS Region is a separate geographic area in which AWS clusters data centers ([docs](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.RegionsAndAvailabilityZones.html)).
|
||||
|
||||
As AWS can have instances in multiple regions, we need to know the region the service you want reach belongs to.
|
||||
|
||||
Note that the AWS Region is the only required parameter when configuring a connection. When connecting to the
|
||||
services programmatically, there are different ways in which we can extract and use the rest of AWS configurations.
|
||||
|
||||
You can find further information about configuring your credentials [here](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html#configuring-credentials).
|
||||
|
||||
|
||||
**endPointURL**: To connect programmatically to an AWS service, you use an endpoint. An *endpoint* is the URL of the
|
||||
entry point for an AWS web service. The AWS SDKs and the AWS Command Line Interface (AWS CLI) automatically use the
|
||||
default endpoint for each service in an AWS Region. But you can specify an alternate endpoint for your API requests.
|
||||
|
||||
Find more information on [AWS service endpoints](https://docs.aws.amazon.com/general/latest/gr/rande.html).
|
||||
|
||||
|
||||
**profileName**: A named profile is a collection of settings and credentials that you can apply to a AWS CLI command.
|
||||
When you specify a profile to run a command, the settings and credentials are used to run that command.
|
||||
Multiple named profiles can be stored in the config and credentials files.
|
||||
|
||||
You can inform this field if you'd like to use a profile other than `default`.
|
||||
|
||||
Find here more information about [Named profiles for the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-profiles.html).
|
||||
|
||||
|
||||
**assumeRoleArn**: Typically, you use `AssumeRole` within your account or for cross-account access. In this field you'll set the
|
||||
`ARN` (Amazon Resource Name) of the policy of the other account.
|
||||
|
||||
A user who wants to access a role in a different account must also have permissions that are delegated from the account
|
||||
administrator. The administrator must attach a policy that allows the user to call `AssumeRole` for the `ARN` of the role in the other account.
|
||||
|
||||
This is a required field if you'd like to `AssumeRole`.
|
||||
|
||||
Find more information on [AssumeRole](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html).
|
||||
|
||||
**assumeRoleSessionName**: An identifier for the assumed role session. Use the role session name to uniquely identify a session when the same role
|
||||
is assumed by different principals or for different reasons.
|
||||
|
||||
By default, we'll use the name `OpenMetadataSession`.
|
||||
|
||||
Find more information about the [Role Session Name](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html#:~:text=An%20identifier%20for%20the%20assumed%20role%20session.).
|
||||
|
||||
|
||||
**assumeRoleSourceIdentity**: The source identity specified by the principal that is calling the `AssumeRole` operation. You can use source identity
|
||||
information in AWS CloudTrail logs to determine who took actions with a role.
|
||||
|
||||
Find more information about [Source Identity](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html#:~:text=Required%3A%20No-,SourceIdentity,-The%20source%20identity).
|
||||
|
||||
{% codeInfo srNumber=4 %}
|
||||
|
||||
|
||||
**hostPort**: Enter the fully qualified hostname and port number for your Postgres deployment in the Host and Port field.
|
||||
|
||||
{% /codeInfo %}
|
||||
|
||||
{% codeInfo srNumber=4 %}
|
||||
{% codeInfo srNumber=5 %}
|
||||
|
||||
**database**: Initial Postgres database to connect to. If you want to ingest all databases, set ingestAllDatabases to true.
|
||||
|
||||
{% /codeInfo %}
|
||||
|
||||
{% codeInfo srNumber=5 %}
|
||||
{% codeInfo srNumber=6 %}
|
||||
|
||||
**ingestAllDatabases**: Ingest data from all databases in Postgres. You can use databaseFilterPattern on top of this.
|
||||
|
||||
@ -130,7 +199,7 @@ This is a sample config for Postgres:
|
||||
|
||||
#### Source Configuration - Source Config
|
||||
|
||||
{% codeInfo srNumber=8 %}
|
||||
{% codeInfo srNumber=9 %}
|
||||
|
||||
The `sourceConfig` is defined [here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/metadataIngestion/databaseServiceMetadataPipeline.json):
|
||||
|
||||
@ -146,7 +215,7 @@ The `sourceConfig` is defined [here](https://github.com/open-metadata/OpenMetada
|
||||
|
||||
#### Sink Configuration
|
||||
|
||||
{% codeInfo srNumber=9 %}
|
||||
{% codeInfo srNumber=10 %}
|
||||
|
||||
To send the metadata to OpenMetadata, it needs to be specified as `type: metadata-rest`.
|
||||
|
||||
@ -154,7 +223,7 @@ To send the metadata to OpenMetadata, it needs to be specified as `type: metadat
|
||||
|
||||
#### Workflow Configuration
|
||||
|
||||
{% codeInfo srNumber=10 %}
|
||||
{% codeInfo srNumber=11 %}
|
||||
|
||||
The main property here is the `openMetadataServerConfig`, where you can define the host and security provider of your OpenMetadata installation.
|
||||
|
||||
@ -164,13 +233,13 @@ For a simple, local installation using our docker containers, this looks like:
|
||||
|
||||
#### Advanced Configuration
|
||||
|
||||
{% codeInfo srNumber=6 %}
|
||||
{% codeInfo srNumber=7 %}
|
||||
|
||||
**Connection Options (Optional)**: Enter the details for any additional connection options that can be sent to Athena during the connection. These details must be added as Key-Value pairs.
|
||||
|
||||
{% /codeInfo %}
|
||||
|
||||
{% codeInfo srNumber=7 %}
|
||||
{% codeInfo srNumber=8 %}
|
||||
|
||||
**Connection Arguments (Optional)**: Enter the details for any additional connection arguments such as security or protocol configs that can be sent to Athena during the connection. These details must be added as Key-Value pairs.
|
||||
|
||||
@ -190,31 +259,39 @@ source:
|
||||
config:
|
||||
type: Postgres
|
||||
```
|
||||
```yaml {% srNumber=6 %}
|
||||
```yaml {% srNumber=1 %}
|
||||
username: username
|
||||
```
|
||||
```yaml {% srNumber=6 %}
|
||||
password: password
|
||||
```yaml {% srNumber=2 %}
|
||||
authType:
|
||||
password: <password>
|
||||
```
|
||||
```yaml {% srNumber=6 %}
|
||||
```yaml {% srNumber=3 %}
|
||||
authType:
|
||||
awsConfig:
|
||||
awsAccessKeyId: access key id
|
||||
awsSecretAccessKey: access secret key
|
||||
awsRegion: aws region name
|
||||
```
|
||||
```yaml {% srNumber=4 %}
|
||||
hostPort: localhost:5432
|
||||
```
|
||||
```yaml {% srNumber=6 %}
|
||||
```yaml {% srNumber=5 %}
|
||||
database: database
|
||||
```
|
||||
```yaml {% srNumber=6 %}
|
||||
ingestAllDatabases: true
|
||||
```
|
||||
```yaml {% srNumber=6 %}
|
||||
```yaml {% srNumber=7 %}
|
||||
# connectionOptions:
|
||||
# key: value
|
||||
```
|
||||
```yaml {% srNumber=7 %}
|
||||
```yaml {% srNumber=8 %}
|
||||
# connectionArguments:
|
||||
# key: value
|
||||
```
|
||||
|
||||
```yaml {% srNumber=8 %}
|
||||
```yaml {% srNumber=9 %}
|
||||
sourceConfig:
|
||||
config:
|
||||
type: DatabaseMetadata
|
||||
@ -245,13 +322,13 @@ source:
|
||||
# - table4
|
||||
```
|
||||
|
||||
```yaml {% srNumber=9 %}
|
||||
```yaml {% srNumber=10 %}
|
||||
sink:
|
||||
type: metadata-rest
|
||||
config: {}
|
||||
```
|
||||
|
||||
```yaml {% srNumber=10 %}
|
||||
```yaml {% srNumber=11 %}
|
||||
workflowConfig:
|
||||
openMetadataServerConfig:
|
||||
hostPort: "http://localhost:8585/api"
|
||||
|
||||
@ -205,7 +205,69 @@ desired.
|
||||
#### Connection Options
|
||||
|
||||
- **Username**: Specify the User to connect to Postgres. It should have enough privileges to read all the metadata.
|
||||
- **Password**: Password to connect to Postgres.
|
||||
- **Auth Type**: Basic Auth or IAM based auth to connect to instances / cloud rds.
|
||||
- **Basic Auth**:
|
||||
- **Password**: Password to connect to Postgres.
|
||||
- **IAM Based Auth**:
|
||||
- **AWS Access Key ID** & **AWS Secret Access Key**: When you interact with AWS, you specify your AWS security credentials to verify who you are and whether you have
|
||||
permission to access the resources that you are requesting. AWS uses the security credentials to authenticate and
|
||||
authorize your requests ([docs](https://docs.aws.amazon.com/IAM/latest/UserGuide/security-creds.html)).
|
||||
|
||||
Access keys consist of two parts: An **access key ID** (for example, `AKIAIOSFODNN7EXAMPLE`), and a **secret access key** (for example, `wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY`).
|
||||
|
||||
You must use both the access key ID and secret access key together to authenticate your requests.
|
||||
|
||||
You can find further information on how to manage your access keys [here](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html).
|
||||
|
||||
- **AWS Region**: Each AWS Region is a separate geographic area in which AWS clusters data centers ([docs](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.RegionsAndAvailabilityZones.html)).
|
||||
|
||||
As AWS can have instances in multiple regions, we need to know the region the service you want reach belongs to.
|
||||
|
||||
Note that the AWS Region is the only required parameter when configuring a connection. When connecting to the
|
||||
services programmatically, there are different ways in which we can extract and use the rest of AWS configurations.
|
||||
|
||||
You can find further information about configuring your credentials [here](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html#configuring-credentials).
|
||||
|
||||
- **AWS Session Token (optional)**: If you are using temporary credentials to access your services, you will need to inform the AWS Access Key ID
|
||||
and AWS Secrets Access Key. Also, these will include an AWS Session Token.
|
||||
|
||||
You can find more information on [Using temporary credentials with AWS resources](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_use-resources.html).
|
||||
|
||||
- **Endpoint URL (optional)**: To connect programmatically to an AWS service, you use an endpoint. An *endpoint* is the URL of the
|
||||
entry point for an AWS web service. The AWS SDKs and the AWS Command Line Interface (AWS CLI) automatically use the
|
||||
default endpoint for each service in an AWS Region. But you can specify an alternate endpoint for your API requests.
|
||||
|
||||
Find more information on [AWS service endpoints](https://docs.aws.amazon.com/general/latest/gr/rande.html).
|
||||
|
||||
- **Profile Name**: A named profile is a collection of settings and credentials that you can apply to a AWS CLI command.
|
||||
When you specify a profile to run a command, the settings and credentials are used to run that command.
|
||||
Multiple named profiles can be stored in the config and credentials files.
|
||||
|
||||
You can inform this field if you'd like to use a profile other than `default`.
|
||||
|
||||
Find here more information about [Named profiles for the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-profiles.html).
|
||||
|
||||
- **Assume Role Arn**: Typically, you use `AssumeRole` within your account or for cross-account access. In this field you'll set the
|
||||
`ARN` (Amazon Resource Name) of the policy of the other account.
|
||||
|
||||
A user who wants to access a role in a different account must also have permissions that are delegated from the account
|
||||
administrator. The administrator must attach a policy that allows the user to call `AssumeRole` for the `ARN` of the role in the other account.
|
||||
|
||||
This is a required field if you'd like to `AssumeRole`.
|
||||
|
||||
Find more information on [AssumeRole](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html).
|
||||
|
||||
- **Assume Role Session Name**: An identifier for the assumed role session. Use the role session name to uniquely identify a session when the same role
|
||||
is assumed by different principals or for different reasons.
|
||||
|
||||
By default, we'll use the name `OpenMetadataSession`.
|
||||
|
||||
Find more information about the [Role Session Name](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html#:~:text=An%20identifier%20for%20the%20assumed%20role%20session.).
|
||||
|
||||
- **Assume Role Source Identity**: The source identity specified by the principal that is calling the `AssumeRole` operation. You can use source identity
|
||||
information in AWS CloudTrail logs to determine who took actions with a role.
|
||||
|
||||
Find more information about [Source Identity](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html#:~:text=Required%3A%20No-,SourceIdentity,-The%20source%20identity).
|
||||
- **Host and Port**: Enter the fully qualified hostname and port number for your Postgres deployment in the Host and Port field.
|
||||
- **Connection Options (Optional)**: Enter the details for any additional connection options that can be sent to Postgres during the connection. These details must be added as Key-Value pairs.
|
||||
- **Connection Arguments (Optional)**: Enter the details for any additional connection arguments such as security or protocol configs that can be sent to Postgres during the connection. These details must be added as Key-Value pairs.
|
||||
|
||||
@ -76,6 +76,10 @@ source:
|
||||
authOptions: <auth options>
|
||||
authMechanism: PLAIN # NOSASL, PLAIN, GSSAPI, LDAP, JWT
|
||||
hostPort: <impala connection host & port>
|
||||
# kerberosServiceName: KerberosServiceName
|
||||
# databaseSchema: Database Schema of the data source
|
||||
# databaseName: Optional name to give to the database in OpenMetadata.
|
||||
# useSSL: true / false
|
||||
sourceConfig:
|
||||
config:
|
||||
type: DatabaseMetadata
|
||||
@ -327,6 +331,10 @@ source:
|
||||
authOptions: <auth options>
|
||||
authMechanism: PLAIN # NOSASL, PLAIN, GSSAPI, LDAP, JWT
|
||||
hostPort: <impala connection host & port>
|
||||
# kerberosServiceName: KerberosServiceName
|
||||
# databaseSchema: Database Schema of the data source
|
||||
# databaseName: Optional name to give to the database in OpenMetadata.
|
||||
# useSSL: true / false
|
||||
sourceConfig:
|
||||
config:
|
||||
type: Profiler
|
||||
@ -444,4 +452,4 @@ Note how instead of running `ingest`, we are using the `profile` command to sele
|
||||
|
||||
## dbt Integration
|
||||
|
||||
You can learn more about how to ingest dbt models' definitions and their lineage [here](/connectors/ingestion/workflows/dbt).
|
||||
You can learn more about how to ingest dbt models' definitions and their lineage [here](/connectors/ingestion/workflows/dbt).
|
||||
@ -83,17 +83,89 @@ This is a sample config for MySQL:
|
||||
|
||||
{% codeInfo srNumber=2 %}
|
||||
|
||||
**password**: Password to connect to MySQL.
|
||||
**authType**: Choose from basic auth and IAM based auth.
|
||||
#### Basic Auth
|
||||
|
||||
**password**: Password comes under Basic Auth type.
|
||||
|
||||
{% /codeInfo %}
|
||||
|
||||
{% codeInfo srNumber=3 %}
|
||||
#### IAM BASED Auth
|
||||
|
||||
- **awsAccessKeyId** & **awsSecretAccessKey**: When you interact with AWS, you specify your AWS security credentials to verify who you are and whether you have
|
||||
permission to access the resources that you are requesting. AWS uses the security credentials to authenticate and
|
||||
authorize your requests ([docs](https://docs.aws.amazon.com/IAM/latest/UserGuide/security-creds.html)).
|
||||
|
||||
Access keys consist of two parts: An **access key ID** (for example, `AKIAIOSFODNN7EXAMPLE`), and a **secret access key** (for example, `wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY`).
|
||||
|
||||
You must use both the access key ID and secret access key together to authenticate your requests.
|
||||
|
||||
You can find further information on how to manage your access keys [here](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html).
|
||||
|
||||
**awsSessionToken**: If you are using temporary credentials to access your services, you will need to inform the AWS Access Key ID
|
||||
and AWS Secrets Access Key. Also, these will include an AWS Session Token.
|
||||
|
||||
|
||||
**awsRegion**: Each AWS Region is a separate geographic area in which AWS clusters data centers ([docs](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.RegionsAndAvailabilityZones.html)).
|
||||
|
||||
As AWS can have instances in multiple regions, we need to know the region the service you want reach belongs to.
|
||||
|
||||
Note that the AWS Region is the only required parameter when configuring a connection. When connecting to the
|
||||
services programmatically, there are different ways in which we can extract and use the rest of AWS configurations.
|
||||
|
||||
You can find further information about configuring your credentials [here](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html#configuring-credentials).
|
||||
|
||||
|
||||
**endPointURL**: To connect programmatically to an AWS service, you use an endpoint. An *endpoint* is the URL of the
|
||||
entry point for an AWS web service. The AWS SDKs and the AWS Command Line Interface (AWS CLI) automatically use the
|
||||
default endpoint for each service in an AWS Region. But you can specify an alternate endpoint for your API requests.
|
||||
|
||||
Find more information on [AWS service endpoints](https://docs.aws.amazon.com/general/latest/gr/rande.html).
|
||||
|
||||
|
||||
**profileName**: A named profile is a collection of settings and credentials that you can apply to a AWS CLI command.
|
||||
When you specify a profile to run a command, the settings and credentials are used to run that command.
|
||||
Multiple named profiles can be stored in the config and credentials files.
|
||||
|
||||
You can inform this field if you'd like to use a profile other than `default`.
|
||||
|
||||
Find here more information about [Named profiles for the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-profiles.html).
|
||||
|
||||
|
||||
**assumeRoleArn**: Typically, you use `AssumeRole` within your account or for cross-account access. In this field you'll set the
|
||||
`ARN` (Amazon Resource Name) of the policy of the other account.
|
||||
|
||||
A user who wants to access a role in a different account must also have permissions that are delegated from the account
|
||||
administrator. The administrator must attach a policy that allows the user to call `AssumeRole` for the `ARN` of the role in the other account.
|
||||
|
||||
This is a required field if you'd like to `AssumeRole`.
|
||||
|
||||
Find more information on [AssumeRole](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html).
|
||||
|
||||
**assumeRoleSessionName**: An identifier for the assumed role session. Use the role session name to uniquely identify a session when the same role
|
||||
is assumed by different principals or for different reasons.
|
||||
|
||||
By default, we'll use the name `OpenMetadataSession`.
|
||||
|
||||
Find more information about the [Role Session Name](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html#:~:text=An%20identifier%20for%20the%20assumed%20role%20session.).
|
||||
|
||||
|
||||
**assumeRoleSourceIdentity**: The source identity specified by the principal that is calling the `AssumeRole` operation. You can use source identity
|
||||
information in AWS CloudTrail logs to determine who took actions with a role.
|
||||
|
||||
Find more information about [Source Identity](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html#:~:text=Required%3A%20No-,SourceIdentity,-The%20source%20identity).
|
||||
|
||||
{% /codeInfo %}
|
||||
|
||||
|
||||
{% codeInfo srNumber=4 %}
|
||||
|
||||
**Host and Port**: Enter the fully qualified hostname and port number for your MySQL deployment in the Host and Port field.
|
||||
|
||||
{% /codeInfo %}
|
||||
|
||||
{% codeInfo srNumber=4 %}
|
||||
{% codeInfo srNumber=5 %}
|
||||
|
||||
**databaseSchema**: databaseSchema of the data source. This is optional parameter, if you would like to restrict the metadata reading to a single databaseSchema. When left blank, OpenMetadata Ingestion attempts to scan all the databaseSchema.
|
||||
|
||||
@ -101,7 +173,7 @@ This is a sample config for MySQL:
|
||||
|
||||
#### Source Configuration - Source Config
|
||||
|
||||
{% codeInfo srNumber=7 %}
|
||||
{% codeInfo srNumber=8 %}
|
||||
|
||||
The `sourceConfig` is defined [here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/metadataIngestion/databaseServiceMetadataPipeline.json):
|
||||
|
||||
@ -118,7 +190,7 @@ The `sourceConfig` is defined [here](https://github.com/open-metadata/OpenMetada
|
||||
|
||||
#### Sink Configuration
|
||||
|
||||
{% codeInfo srNumber=8 %}
|
||||
{% codeInfo srNumber=9 %}
|
||||
|
||||
To send the metadata to OpenMetadata, it needs to be specified as `type: metadata-rest`.
|
||||
|
||||
@ -126,7 +198,7 @@ To send the metadata to OpenMetadata, it needs to be specified as `type: metadat
|
||||
|
||||
#### Workflow Configuration
|
||||
|
||||
{% codeInfo srNumber=9 %}
|
||||
{% codeInfo srNumber=10 %}
|
||||
|
||||
The main property here is the `openMetadataServerConfig`, where you can define the host and security provider of your OpenMetadata installation.
|
||||
|
||||
@ -136,13 +208,13 @@ For a simple, local installation using our docker containers, this looks like:
|
||||
|
||||
#### Advanced Configuration
|
||||
|
||||
{% codeInfo srNumber=5 %}
|
||||
{% codeInfo srNumber=6 %}
|
||||
|
||||
**Connection Options (Optional)**: Enter the details for any additional connection options that can be sent to Athena during the connection. These details must be added as Key-Value pairs.
|
||||
|
||||
{% /codeInfo %}
|
||||
|
||||
{% codeInfo srNumber=6 %}
|
||||
{% codeInfo srNumber=7 %}
|
||||
|
||||
**Connection Arguments (Optional)**: Enter the details for any additional connection arguments such as security or protocol configs that can be sent to Athena during the connection. These details must be added as Key-Value pairs.
|
||||
|
||||
@ -166,23 +238,32 @@ source:
|
||||
username: <username>
|
||||
```
|
||||
```yaml {% srNumber=2 %}
|
||||
password: <password>
|
||||
authType:
|
||||
password: <password>
|
||||
```
|
||||
```yaml {% srNumber=3 %}
|
||||
hostPort: <hostPort>
|
||||
authType:
|
||||
awsConfig:
|
||||
awsAccessKeyId: access key id
|
||||
awsSecretAccessKey: access secret key
|
||||
awsRegion: aws region name
|
||||
```
|
||||
```yaml {% srNumber=4 %}
|
||||
databaseSchema: schema
|
||||
hostPort: <hostPort>
|
||||
```
|
||||
```yaml {% srNumber=5 %}
|
||||
databaseSchema: schema
|
||||
```
|
||||
|
||||
```yaml {% srNumber=6 %}
|
||||
# connectionOptions:
|
||||
# key: value
|
||||
```
|
||||
```yaml {% srNumber=6 %}
|
||||
```yaml {% srNumber=7 %}
|
||||
# connectionArguments:
|
||||
# key: value
|
||||
```
|
||||
```yaml {% srNumber=7 %}
|
||||
```yaml {% srNumber=8 %}
|
||||
sourceConfig:
|
||||
config:
|
||||
type: DatabaseMetadata
|
||||
@ -213,13 +294,13 @@ source:
|
||||
# - table4
|
||||
```
|
||||
|
||||
```yaml {% srNumber=8 %}
|
||||
```yaml {% srNumber=9 %}
|
||||
sink:
|
||||
type: metadata-rest
|
||||
config: {}
|
||||
```
|
||||
|
||||
```yaml {% srNumber=9 %}
|
||||
```yaml {% srNumber=10 %}
|
||||
workflowConfig:
|
||||
# loggerLevel: DEBUG # DEBUG, INFO, WARN or ERROR
|
||||
openMetadataServerConfig:
|
||||
|
||||
@ -201,7 +201,69 @@ desired.
|
||||
#### Connection Options
|
||||
|
||||
- **Username**: Specify the User to connect to MySQL. It should have enough privileges to read all the metadata.
|
||||
- **Password**: Password to connect to MySQL.
|
||||
- **Auth Type**: Basic Auth or IAM based auth to connect to instances / cloud rds.
|
||||
- **Basic Auth**:
|
||||
- **Password**: Password to connect to MySQL.
|
||||
- **IAM Based Auth**:
|
||||
- **AWS Access Key ID** & **AWS Secret Access Key**: When you interact with AWS, you specify your AWS security credentials to verify who you are and whether you have
|
||||
permission to access the resources that you are requesting. AWS uses the security credentials to authenticate and
|
||||
authorize your requests ([docs](https://docs.aws.amazon.com/IAM/latest/UserGuide/security-creds.html)).
|
||||
|
||||
Access keys consist of two parts: An **access key ID** (for example, `AKIAIOSFODNN7EXAMPLE`), and a **secret access key** (for example, `wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY`).
|
||||
|
||||
You must use both the access key ID and secret access key together to authenticate your requests.
|
||||
|
||||
You can find further information on how to manage your access keys [here](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html).
|
||||
|
||||
- **AWS Region**: Each AWS Region is a separate geographic area in which AWS clusters data centers ([docs](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.RegionsAndAvailabilityZones.html)).
|
||||
|
||||
As AWS can have instances in multiple regions, we need to know the region the service you want reach belongs to.
|
||||
|
||||
Note that the AWS Region is the only required parameter when configuring a connection. When connecting to the
|
||||
services programmatically, there are different ways in which we can extract and use the rest of AWS configurations.
|
||||
|
||||
You can find further information about configuring your credentials [here](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html#configuring-credentials).
|
||||
|
||||
- **AWS Session Token (optional)**: If you are using temporary credentials to access your services, you will need to inform the AWS Access Key ID
|
||||
and AWS Secrets Access Key. Also, these will include an AWS Session Token.
|
||||
|
||||
You can find more information on [Using temporary credentials with AWS resources](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_use-resources.html).
|
||||
|
||||
- **Endpoint URL (optional)**: To connect programmatically to an AWS service, you use an endpoint. An *endpoint* is the URL of the
|
||||
entry point for an AWS web service. The AWS SDKs and the AWS Command Line Interface (AWS CLI) automatically use the
|
||||
default endpoint for each service in an AWS Region. But you can specify an alternate endpoint for your API requests.
|
||||
|
||||
Find more information on [AWS service endpoints](https://docs.aws.amazon.com/general/latest/gr/rande.html).
|
||||
|
||||
- **Profile Name**: A named profile is a collection of settings and credentials that you can apply to a AWS CLI command.
|
||||
When you specify a profile to run a command, the settings and credentials are used to run that command.
|
||||
Multiple named profiles can be stored in the config and credentials files.
|
||||
|
||||
You can inform this field if you'd like to use a profile other than `default`.
|
||||
|
||||
Find here more information about [Named profiles for the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-profiles.html).
|
||||
|
||||
- **Assume Role Arn**: Typically, you use `AssumeRole` within your account or for cross-account access. In this field you'll set the
|
||||
`ARN` (Amazon Resource Name) of the policy of the other account.
|
||||
|
||||
A user who wants to access a role in a different account must also have permissions that are delegated from the account
|
||||
administrator. The administrator must attach a policy that allows the user to call `AssumeRole` for the `ARN` of the role in the other account.
|
||||
|
||||
This is a required field if you'd like to `AssumeRole`.
|
||||
|
||||
Find more information on [AssumeRole](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html).
|
||||
|
||||
- **Assume Role Session Name**: An identifier for the assumed role session. Use the role session name to uniquely identify a session when the same role
|
||||
is assumed by different principals or for different reasons.
|
||||
|
||||
By default, we'll use the name `OpenMetadataSession`.
|
||||
|
||||
Find more information about the [Role Session Name](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html#:~:text=An%20identifier%20for%20the%20assumed%20role%20session.).
|
||||
|
||||
- **Assume Role Source Identity**: The source identity specified by the principal that is calling the `AssumeRole` operation. You can use source identity
|
||||
information in AWS CloudTrail logs to determine who took actions with a role.
|
||||
|
||||
Find more information about [Source Identity](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html#:~:text=Required%3A%20No-,SourceIdentity,-The%20source%20identity).
|
||||
- **Host and Port**: Enter the fully qualified hostname and port number for your MySQL deployment in the Host and Port field.
|
||||
- **databaseName**: Optional name to give to the database in OpenMetadata. If left blank, we will use default as the database name.
|
||||
- **databaseSchema**: databaseSchema of the data source. This is optional parameter, if you would like to restrict the metadata reading to a single databaseSchema. When left blank, OpenMetadata Ingestion attempts to scan all the databaseSchema.
|
||||
|
||||
@ -103,26 +103,95 @@ This is a sample config for Postgres:
|
||||
**username**: Specify the User to connect to Postgres. It should have enough privileges to read all the metadata.
|
||||
|
||||
{% /codeInfo %}
|
||||
|
||||
{% codeInfo srNumber=2 %}
|
||||
|
||||
**password**: Password to connect to Postgres.
|
||||
**authType**: Choose from basic auth and IAM based auth.
|
||||
#### Basic Auth
|
||||
|
||||
**password**: Password comes under Basic Auth type.
|
||||
|
||||
{% /codeInfo %}
|
||||
|
||||
{% codeInfo srNumber=3 %}
|
||||
#### IAM BASED Auth
|
||||
|
||||
- **awsAccessKeyId** & **awsSecretAccessKey**: When you interact with AWS, you specify your AWS security credentials to verify who you are and whether you have
|
||||
permission to access the resources that you are requesting. AWS uses the security credentials to authenticate and
|
||||
authorize your requests ([docs](https://docs.aws.amazon.com/IAM/latest/UserGuide/security-creds.html)).
|
||||
|
||||
Access keys consist of two parts: An **access key ID** (for example, `AKIAIOSFODNN7EXAMPLE`), and a **secret access key** (for example, `wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY`).
|
||||
|
||||
You must use both the access key ID and secret access key together to authenticate your requests.
|
||||
|
||||
You can find further information on how to manage your access keys [here](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html).
|
||||
|
||||
**awsSessionToken**: If you are using temporary credentials to access your services, you will need to inform the AWS Access Key ID
|
||||
and AWS Secrets Access Key. Also, these will include an AWS Session Token.
|
||||
|
||||
|
||||
**awsRegion**: Each AWS Region is a separate geographic area in which AWS clusters data centers ([docs](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.RegionsAndAvailabilityZones.html)).
|
||||
|
||||
As AWS can have instances in multiple regions, we need to know the region the service you want reach belongs to.
|
||||
|
||||
Note that the AWS Region is the only required parameter when configuring a connection. When connecting to the
|
||||
services programmatically, there are different ways in which we can extract and use the rest of AWS configurations.
|
||||
|
||||
You can find further information about configuring your credentials [here](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html#configuring-credentials).
|
||||
|
||||
|
||||
**endPointURL**: To connect programmatically to an AWS service, you use an endpoint. An *endpoint* is the URL of the
|
||||
entry point for an AWS web service. The AWS SDKs and the AWS Command Line Interface (AWS CLI) automatically use the
|
||||
default endpoint for each service in an AWS Region. But you can specify an alternate endpoint for your API requests.
|
||||
|
||||
Find more information on [AWS service endpoints](https://docs.aws.amazon.com/general/latest/gr/rande.html).
|
||||
|
||||
|
||||
**profileName**: A named profile is a collection of settings and credentials that you can apply to a AWS CLI command.
|
||||
When you specify a profile to run a command, the settings and credentials are used to run that command.
|
||||
Multiple named profiles can be stored in the config and credentials files.
|
||||
|
||||
You can inform this field if you'd like to use a profile other than `default`.
|
||||
|
||||
Find here more information about [Named profiles for the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-profiles.html).
|
||||
|
||||
|
||||
**assumeRoleArn**: Typically, you use `AssumeRole` within your account or for cross-account access. In this field you'll set the
|
||||
`ARN` (Amazon Resource Name) of the policy of the other account.
|
||||
|
||||
A user who wants to access a role in a different account must also have permissions that are delegated from the account
|
||||
administrator. The administrator must attach a policy that allows the user to call `AssumeRole` for the `ARN` of the role in the other account.
|
||||
|
||||
This is a required field if you'd like to `AssumeRole`.
|
||||
|
||||
Find more information on [AssumeRole](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html).
|
||||
|
||||
**assumeRoleSessionName**: An identifier for the assumed role session. Use the role session name to uniquely identify a session when the same role
|
||||
is assumed by different principals or for different reasons.
|
||||
|
||||
By default, we'll use the name `OpenMetadataSession`.
|
||||
|
||||
Find more information about the [Role Session Name](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html#:~:text=An%20identifier%20for%20the%20assumed%20role%20session.).
|
||||
|
||||
|
||||
**assumeRoleSourceIdentity**: The source identity specified by the principal that is calling the `AssumeRole` operation. You can use source identity
|
||||
information in AWS CloudTrail logs to determine who took actions with a role.
|
||||
|
||||
Find more information about [Source Identity](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html#:~:text=Required%3A%20No-,SourceIdentity,-The%20source%20identity).
|
||||
|
||||
{% codeInfo srNumber=4 %}
|
||||
|
||||
|
||||
**hostPort**: Enter the fully qualified hostname and port number for your Postgres deployment in the Host and Port field.
|
||||
|
||||
{% /codeInfo %}
|
||||
|
||||
{% codeInfo srNumber=4 %}
|
||||
{% codeInfo srNumber=5 %}
|
||||
|
||||
**database**: Initial Postgres database to connect to. If you want to ingest all databases, set ingestAllDatabases to true.
|
||||
|
||||
{% /codeInfo %}
|
||||
|
||||
{% codeInfo srNumber=5 %}
|
||||
{% codeInfo srNumber=6 %}
|
||||
|
||||
**ingestAllDatabases**: Ingest data from all databases in Postgres. You can use databaseFilterPattern on top of this.
|
||||
|
||||
@ -130,7 +199,7 @@ This is a sample config for Postgres:
|
||||
|
||||
#### Source Configuration - Source Config
|
||||
|
||||
{% codeInfo srNumber=8 %}
|
||||
{% codeInfo srNumber=9 %}
|
||||
|
||||
The `sourceConfig` is defined [here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/metadataIngestion/databaseServiceMetadataPipeline.json):
|
||||
|
||||
@ -146,7 +215,7 @@ The `sourceConfig` is defined [here](https://github.com/open-metadata/OpenMetada
|
||||
|
||||
#### Sink Configuration
|
||||
|
||||
{% codeInfo srNumber=9 %}
|
||||
{% codeInfo srNumber=10 %}
|
||||
|
||||
To send the metadata to OpenMetadata, it needs to be specified as `type: metadata-rest`.
|
||||
|
||||
@ -154,7 +223,7 @@ To send the metadata to OpenMetadata, it needs to be specified as `type: metadat
|
||||
|
||||
#### Workflow Configuration
|
||||
|
||||
{% codeInfo srNumber=10 %}
|
||||
{% codeInfo srNumber=11 %}
|
||||
|
||||
The main property here is the `openMetadataServerConfig`, where you can define the host and security provider of your OpenMetadata installation.
|
||||
|
||||
@ -164,13 +233,13 @@ For a simple, local installation using our docker containers, this looks like:
|
||||
|
||||
#### Advanced Configuration
|
||||
|
||||
{% codeInfo srNumber=6 %}
|
||||
{% codeInfo srNumber=7 %}
|
||||
|
||||
**Connection Options (Optional)**: Enter the details for any additional connection options that can be sent to Athena during the connection. These details must be added as Key-Value pairs.
|
||||
|
||||
{% /codeInfo %}
|
||||
|
||||
{% codeInfo srNumber=7 %}
|
||||
{% codeInfo srNumber=8 %}
|
||||
|
||||
**Connection Arguments (Optional)**: Enter the details for any additional connection arguments such as security or protocol configs that can be sent to Athena during the connection. These details must be added as Key-Value pairs.
|
||||
|
||||
@ -190,31 +259,39 @@ source:
|
||||
config:
|
||||
type: Postgres
|
||||
```
|
||||
```yaml {% srNumber=6 %}
|
||||
```yaml {% srNumber=1 %}
|
||||
username: username
|
||||
```
|
||||
```yaml {% srNumber=6 %}
|
||||
password: password
|
||||
```yaml {% srNumber=2 %}
|
||||
authType:
|
||||
password: <password>
|
||||
```
|
||||
```yaml {% srNumber=6 %}
|
||||
```yaml {% srNumber=3 %}
|
||||
authType:
|
||||
awsConfig:
|
||||
awsAccessKeyId: access key id
|
||||
awsSecretAccessKey: access secret key
|
||||
awsRegion: aws region name
|
||||
```
|
||||
```yaml {% srNumber=4 %}
|
||||
hostPort: localhost:5432
|
||||
```
|
||||
```yaml {% srNumber=6 %}
|
||||
```yaml {% srNumber=5 %}
|
||||
database: database
|
||||
```
|
||||
```yaml {% srNumber=6 %}
|
||||
ingestAllDatabases: true
|
||||
```
|
||||
```yaml {% srNumber=6 %}
|
||||
```yaml {% srNumber=7 %}
|
||||
# connectionOptions:
|
||||
# key: value
|
||||
```
|
||||
```yaml {% srNumber=7 %}
|
||||
```yaml {% srNumber=8 %}
|
||||
# connectionArguments:
|
||||
# key: value
|
||||
```
|
||||
|
||||
```yaml {% srNumber=8 %}
|
||||
```yaml {% srNumber=9 %}
|
||||
sourceConfig:
|
||||
config:
|
||||
type: DatabaseMetadata
|
||||
@ -245,13 +322,13 @@ source:
|
||||
# - table4
|
||||
```
|
||||
|
||||
```yaml {% srNumber=9 %}
|
||||
```yaml {% srNumber=10 %}
|
||||
sink:
|
||||
type: metadata-rest
|
||||
config: {}
|
||||
```
|
||||
|
||||
```yaml {% srNumber=10 %}
|
||||
```yaml {% srNumber=11 %}
|
||||
workflowConfig:
|
||||
openMetadataServerConfig:
|
||||
hostPort: "http://localhost:8585/api"
|
||||
|
||||
@ -205,7 +205,69 @@ desired.
|
||||
#### Connection Options
|
||||
|
||||
- **Username**: Specify the User to connect to Postgres. It should have enough privileges to read all the metadata.
|
||||
- **Password**: Password to connect to Postgres.
|
||||
- **Auth Type**: Basic Auth or IAM based auth to connect to instances / cloud rds.
|
||||
- **Basic Auth**:
|
||||
- **Password**: Password to connect to Postgres.
|
||||
- **IAM Based Auth**:
|
||||
- **AWS Access Key ID** & **AWS Secret Access Key**: When you interact with AWS, you specify your AWS security credentials to verify who you are and whether you have
|
||||
permission to access the resources that you are requesting. AWS uses the security credentials to authenticate and
|
||||
authorize your requests ([docs](https://docs.aws.amazon.com/IAM/latest/UserGuide/security-creds.html)).
|
||||
|
||||
Access keys consist of two parts: An **access key ID** (for example, `AKIAIOSFODNN7EXAMPLE`), and a **secret access key** (for example, `wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY`).
|
||||
|
||||
You must use both the access key ID and secret access key together to authenticate your requests.
|
||||
|
||||
You can find further information on how to manage your access keys [here](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html).
|
||||
|
||||
- **AWS Region**: Each AWS Region is a separate geographic area in which AWS clusters data centers ([docs](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.RegionsAndAvailabilityZones.html)).
|
||||
|
||||
As AWS can have instances in multiple regions, we need to know the region the service you want reach belongs to.
|
||||
|
||||
Note that the AWS Region is the only required parameter when configuring a connection. When connecting to the
|
||||
services programmatically, there are different ways in which we can extract and use the rest of AWS configurations.
|
||||
|
||||
You can find further information about configuring your credentials [here](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html#configuring-credentials).
|
||||
|
||||
- **AWS Session Token (optional)**: If you are using temporary credentials to access your services, you will need to inform the AWS Access Key ID
|
||||
and AWS Secrets Access Key. Also, these will include an AWS Session Token.
|
||||
|
||||
You can find more information on [Using temporary credentials with AWS resources](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_use-resources.html).
|
||||
|
||||
- **Endpoint URL (optional)**: To connect programmatically to an AWS service, you use an endpoint. An *endpoint* is the URL of the
|
||||
entry point for an AWS web service. The AWS SDKs and the AWS Command Line Interface (AWS CLI) automatically use the
|
||||
default endpoint for each service in an AWS Region. But you can specify an alternate endpoint for your API requests.
|
||||
|
||||
Find more information on [AWS service endpoints](https://docs.aws.amazon.com/general/latest/gr/rande.html).
|
||||
|
||||
- **Profile Name**: A named profile is a collection of settings and credentials that you can apply to a AWS CLI command.
|
||||
When you specify a profile to run a command, the settings and credentials are used to run that command.
|
||||
Multiple named profiles can be stored in the config and credentials files.
|
||||
|
||||
You can inform this field if you'd like to use a profile other than `default`.
|
||||
|
||||
Find here more information about [Named profiles for the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-profiles.html).
|
||||
|
||||
- **Assume Role Arn**: Typically, you use `AssumeRole` within your account or for cross-account access. In this field you'll set the
|
||||
`ARN` (Amazon Resource Name) of the policy of the other account.
|
||||
|
||||
A user who wants to access a role in a different account must also have permissions that are delegated from the account
|
||||
administrator. The administrator must attach a policy that allows the user to call `AssumeRole` for the `ARN` of the role in the other account.
|
||||
|
||||
This is a required field if you'd like to `AssumeRole`.
|
||||
|
||||
Find more information on [AssumeRole](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html).
|
||||
|
||||
- **Assume Role Session Name**: An identifier for the assumed role session. Use the role session name to uniquely identify a session when the same role
|
||||
is assumed by different principals or for different reasons.
|
||||
|
||||
By default, we'll use the name `OpenMetadataSession`.
|
||||
|
||||
Find more information about the [Role Session Name](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html#:~:text=An%20identifier%20for%20the%20assumed%20role%20session.).
|
||||
|
||||
- **Assume Role Source Identity**: The source identity specified by the principal that is calling the `AssumeRole` operation. You can use source identity
|
||||
information in AWS CloudTrail logs to determine who took actions with a role.
|
||||
|
||||
Find more information about [Source Identity](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html#:~:text=Required%3A%20No-,SourceIdentity,-The%20source%20identity).
|
||||
- **Host and Port**: Enter the fully qualified hostname and port number for your Postgres deployment in the Host and Port field.
|
||||
- **Connection Options (Optional)**: Enter the details for any additional connection options that can be sent to Postgres during the connection. These details must be added as Key-Value pairs.
|
||||
- **Connection Arguments (Optional)**: Enter the details for any additional connection arguments such as security or protocol configs that can be sent to Postgres during the connection. These details must be added as Key-Value pairs.
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user