mirror of
https://github.com/open-metadata/OpenMetadata.git
synced 2025-10-23 14:54:56 +00:00
Docs: Aws Partial Addition in Connectors (#20342)
This commit is contained in:
parent
6e8fe6ff8f
commit
d1ecf60acb
@ -0,0 +1,94 @@
|
|||||||
|
{% codeInfo srNumber=221 %}
|
||||||
|
|
||||||
|
- **awsAccessKeyId** & **awsSecretAccessKey**: When you interact with AWS, you specify your AWS security credentials to verify who you are and whether you have
|
||||||
|
permission to access the resources that you are requesting. AWS uses the security credentials to authenticate and
|
||||||
|
authorize your requests ([docs](https://docs.aws.amazon.com/IAM/latest/UserGuide/security-creds.html)).
|
||||||
|
|
||||||
|
Access keys consist of two parts: An **access key ID** (for example, `AKIAIOSFODNN7EXAMPLE`), and a **secret access key** (for example, `wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY`).
|
||||||
|
|
||||||
|
You must use both the access key ID and secret access key together to authenticate your requests.
|
||||||
|
|
||||||
|
You can find further information on how to manage your access keys [here](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html).
|
||||||
|
|
||||||
|
{% /codeInfo %}
|
||||||
|
|
||||||
|
{% codeInfo srNumber=222 %}
|
||||||
|
**awsSessionToken**: If you are using temporary credentials to access your services, you will need to inform the AWS Access Key ID
|
||||||
|
and AWS Secrets Access Key. Also, these will include an AWS Session Token.
|
||||||
|
|
||||||
|
{% /codeInfo %}
|
||||||
|
|
||||||
|
{% codeInfo srNumber=223 %}
|
||||||
|
|
||||||
|
**awsRegion**: Each AWS Region is a separate geographic area in which AWS clusters data centers ([docs](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.RegionsAndAvailabilityZones.html)).
|
||||||
|
|
||||||
|
As AWS can have instances in multiple regions, we need to know the region the service you want reach belongs to.
|
||||||
|
|
||||||
|
Note that the AWS Region is the only required parameter when configuring a connection. When connecting to the
|
||||||
|
services programmatically, there are different ways in which we can extract and use the rest of AWS configurations.
|
||||||
|
|
||||||
|
You can find further information about configuring your credentials [here](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html#configuring-credentials).
|
||||||
|
|
||||||
|
{% /codeInfo %}
|
||||||
|
|
||||||
|
{% codeInfo srNumber=224 %}
|
||||||
|
|
||||||
|
**endPointURL**: To connect programmatically to an AWS service, you use an endpoint. An *endpoint* is the URL of the
|
||||||
|
entry point for an AWS web service. The AWS SDKs and the AWS Command Line Interface (AWS CLI) automatically use the
|
||||||
|
default endpoint for each service in an AWS Region. But you can specify an alternate endpoint for your API requests.
|
||||||
|
|
||||||
|
Find more information on [AWS service endpoints](https://docs.aws.amazon.com/general/latest/gr/rande.html).
|
||||||
|
|
||||||
|
{% /codeInfo %}
|
||||||
|
|
||||||
|
{% codeInfo srNumber=225 %}
|
||||||
|
|
||||||
|
**profileName**: A named profile is a collection of settings and credentials that you can apply to a AWS CLI command.
|
||||||
|
When you specify a profile to run a command, the settings and credentials are used to run that command.
|
||||||
|
Multiple named profiles can be stored in the config and credentials files.
|
||||||
|
|
||||||
|
You can inform this field if you'd like to use a profile other than `default`.
|
||||||
|
|
||||||
|
Find here more information about [Named profiles for the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-profiles.html).
|
||||||
|
|
||||||
|
{% /codeInfo %}
|
||||||
|
|
||||||
|
{% codeInfo srNumber=226 %}
|
||||||
|
|
||||||
|
**assumeRoleArn**: Typically, you use `AssumeRole` within your account or for cross-account access. In this field you'll set the
|
||||||
|
`ARN` (Amazon Resource Name) of the policy of the other account.
|
||||||
|
|
||||||
|
A user who wants to access a role in a different account must also have permissions that are delegated from the account
|
||||||
|
administrator. The administrator must attach a policy that allows the user to call `AssumeRole` for the `ARN` of the role in the other account.
|
||||||
|
|
||||||
|
This is a required field if you'd like to `AssumeRole`.
|
||||||
|
|
||||||
|
Find more information on [AssumeRole](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html).
|
||||||
|
|
||||||
|
{%note%}
|
||||||
|
When using Assume Role authentication, ensure you provide the following details:
|
||||||
|
- **AWS Region**: Specify the AWS region for your deployment.
|
||||||
|
- **Assume Role ARN**: Provide the ARN of the role in your AWS account that OpenMetadata will assume.
|
||||||
|
{%/note%}
|
||||||
|
|
||||||
|
{% /codeInfo %}
|
||||||
|
|
||||||
|
{% codeInfo srNumber=227 %}
|
||||||
|
|
||||||
|
**assumeRoleSessionName**: An identifier for the assumed role session. Use the role session name to uniquely identify a session when the same role
|
||||||
|
is assumed by different principals or for different reasons.
|
||||||
|
|
||||||
|
By default, we'll use the name `OpenMetadataSession`.
|
||||||
|
|
||||||
|
Find more information about the [Role Session Name](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html#:~:text=An%20identifier%20for%20the%20assumed%20role%20session.).
|
||||||
|
|
||||||
|
{% /codeInfo %}
|
||||||
|
|
||||||
|
{% codeInfo srNumber=228 %}
|
||||||
|
|
||||||
|
**assumeRoleSourceIdentity**: The source identity specified by the principal that is calling the `AssumeRole` operation. You can use source identity
|
||||||
|
information in AWS CloudTrail logs to determine who took actions with a role.
|
||||||
|
|
||||||
|
Find more information about [Source Identity](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html#:~:text=Required%3A%20No-,SourceIdentity,-The%20source%20identity).
|
||||||
|
|
||||||
|
{% /codeInfo %}
|
@ -0,0 +1,25 @@
|
|||||||
|
```yaml {% srNumber=221 %}
|
||||||
|
awsAccessKeyId: KEY
|
||||||
|
awsSecretAccessKey: SECRET
|
||||||
|
```
|
||||||
|
```yaml {% srNumber=222 %}
|
||||||
|
# awsSessionToken: TOKEN
|
||||||
|
```
|
||||||
|
```yaml {% srNumber=223 %}
|
||||||
|
awsRegion: us-east-2
|
||||||
|
```
|
||||||
|
```yaml {% srNumber=224 %}
|
||||||
|
# endPointURL: https://athena.us-east-2.amazonaws.com/custom
|
||||||
|
```
|
||||||
|
```yaml {% srNumber=225 %}
|
||||||
|
# profileName: profile
|
||||||
|
```
|
||||||
|
```yaml {% srNumber=226 %}
|
||||||
|
# assumeRoleArn: "arn:partition:service:region:account:resource"
|
||||||
|
```
|
||||||
|
```yaml {% srNumber=227 %}
|
||||||
|
# assumeRoleSessionName: session
|
||||||
|
```
|
||||||
|
```yaml {% srNumber=228 %}
|
||||||
|
# assumeRoleSourceIdentity: identity
|
||||||
|
```
|
@ -0,0 +1,94 @@
|
|||||||
|
{% codeInfo srNumber=221 %}
|
||||||
|
|
||||||
|
- **awsAccessKeyId** & **awsSecretAccessKey**: When you interact with AWS, you specify your AWS security credentials to verify who you are and whether you have
|
||||||
|
permission to access the resources that you are requesting. AWS uses the security credentials to authenticate and
|
||||||
|
authorize your requests ([docs](https://docs.aws.amazon.com/IAM/latest/UserGuide/security-creds.html)).
|
||||||
|
|
||||||
|
Access keys consist of two parts: An **access key ID** (for example, `AKIAIOSFODNN7EXAMPLE`), and a **secret access key** (for example, `wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY`).
|
||||||
|
|
||||||
|
You must use both the access key ID and secret access key together to authenticate your requests.
|
||||||
|
|
||||||
|
You can find further information on how to manage your access keys [here](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html).
|
||||||
|
|
||||||
|
{% /codeInfo %}
|
||||||
|
|
||||||
|
{% codeInfo srNumber=222 %}
|
||||||
|
**awsSessionToken**: If you are using temporary credentials to access your services, you will need to inform the AWS Access Key ID
|
||||||
|
and AWS Secrets Access Key. Also, these will include an AWS Session Token.
|
||||||
|
|
||||||
|
{% /codeInfo %}
|
||||||
|
|
||||||
|
{% codeInfo srNumber=223 %}
|
||||||
|
|
||||||
|
**awsRegion**: Each AWS Region is a separate geographic area in which AWS clusters data centers ([docs](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.RegionsAndAvailabilityZones.html)).
|
||||||
|
|
||||||
|
As AWS can have instances in multiple regions, we need to know the region the service you want reach belongs to.
|
||||||
|
|
||||||
|
Note that the AWS Region is the only required parameter when configuring a connection. When connecting to the
|
||||||
|
services programmatically, there are different ways in which we can extract and use the rest of AWS configurations.
|
||||||
|
|
||||||
|
You can find further information about configuring your credentials [here](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html#configuring-credentials).
|
||||||
|
|
||||||
|
{% /codeInfo %}
|
||||||
|
|
||||||
|
{% codeInfo srNumber=224 %}
|
||||||
|
|
||||||
|
**endPointURL**: To connect programmatically to an AWS service, you use an endpoint. An *endpoint* is the URL of the
|
||||||
|
entry point for an AWS web service. The AWS SDKs and the AWS Command Line Interface (AWS CLI) automatically use the
|
||||||
|
default endpoint for each service in an AWS Region. But you can specify an alternate endpoint for your API requests.
|
||||||
|
|
||||||
|
Find more information on [AWS service endpoints](https://docs.aws.amazon.com/general/latest/gr/rande.html).
|
||||||
|
|
||||||
|
{% /codeInfo %}
|
||||||
|
|
||||||
|
{% codeInfo srNumber=225 %}
|
||||||
|
|
||||||
|
**profileName**: A named profile is a collection of settings and credentials that you can apply to a AWS CLI command.
|
||||||
|
When you specify a profile to run a command, the settings and credentials are used to run that command.
|
||||||
|
Multiple named profiles can be stored in the config and credentials files.
|
||||||
|
|
||||||
|
You can inform this field if you'd like to use a profile other than `default`.
|
||||||
|
|
||||||
|
Find here more information about [Named profiles for the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-profiles.html).
|
||||||
|
|
||||||
|
{% /codeInfo %}
|
||||||
|
|
||||||
|
{% codeInfo srNumber=226 %}
|
||||||
|
|
||||||
|
**assumeRoleArn**: Typically, you use `AssumeRole` within your account or for cross-account access. In this field you'll set the
|
||||||
|
`ARN` (Amazon Resource Name) of the policy of the other account.
|
||||||
|
|
||||||
|
A user who wants to access a role in a different account must also have permissions that are delegated from the account
|
||||||
|
administrator. The administrator must attach a policy that allows the user to call `AssumeRole` for the `ARN` of the role in the other account.
|
||||||
|
|
||||||
|
This is a required field if you'd like to `AssumeRole`.
|
||||||
|
|
||||||
|
Find more information on [AssumeRole](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html).
|
||||||
|
|
||||||
|
{%note%}
|
||||||
|
When using Assume Role authentication, ensure you provide the following details:
|
||||||
|
- **AWS Region**: Specify the AWS region for your deployment.
|
||||||
|
- **Assume Role ARN**: Provide the ARN of the role in your AWS account that OpenMetadata will assume.
|
||||||
|
{%/note%}
|
||||||
|
|
||||||
|
{% /codeInfo %}
|
||||||
|
|
||||||
|
{% codeInfo srNumber=227 %}
|
||||||
|
|
||||||
|
**assumeRoleSessionName**: An identifier for the assumed role session. Use the role session name to uniquely identify a session when the same role
|
||||||
|
is assumed by different principals or for different reasons.
|
||||||
|
|
||||||
|
By default, we'll use the name `OpenMetadataSession`.
|
||||||
|
|
||||||
|
Find more information about the [Role Session Name](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html#:~:text=An%20identifier%20for%20the%20assumed%20role%20session.).
|
||||||
|
|
||||||
|
{% /codeInfo %}
|
||||||
|
|
||||||
|
{% codeInfo srNumber=228 %}
|
||||||
|
|
||||||
|
**assumeRoleSourceIdentity**: The source identity specified by the principal that is calling the `AssumeRole` operation. You can use source identity
|
||||||
|
information in AWS CloudTrail logs to determine who took actions with a role.
|
||||||
|
|
||||||
|
Find more information about [Source Identity](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html#:~:text=Required%3A%20No-,SourceIdentity,-The%20source%20identity).
|
||||||
|
|
||||||
|
{% /codeInfo %}
|
@ -0,0 +1,25 @@
|
|||||||
|
```yaml {% srNumber=221 %}
|
||||||
|
awsAccessKeyId: KEY
|
||||||
|
awsSecretAccessKey: SECRET
|
||||||
|
```
|
||||||
|
```yaml {% srNumber=222 %}
|
||||||
|
# awsSessionToken: TOKEN
|
||||||
|
```
|
||||||
|
```yaml {% srNumber=223 %}
|
||||||
|
awsRegion: us-east-2
|
||||||
|
```
|
||||||
|
```yaml {% srNumber=224 %}
|
||||||
|
# endPointURL: https://athena.us-east-2.amazonaws.com/custom
|
||||||
|
```
|
||||||
|
```yaml {% srNumber=225 %}
|
||||||
|
# profileName: profile
|
||||||
|
```
|
||||||
|
```yaml {% srNumber=226 %}
|
||||||
|
# assumeRoleArn: "arn:partition:service:region:account:resource"
|
||||||
|
```
|
||||||
|
```yaml {% srNumber=227 %}
|
||||||
|
# assumeRoleSessionName: session
|
||||||
|
```
|
||||||
|
```yaml {% srNumber=228 %}
|
||||||
|
# assumeRoleSourceIdentity: identity
|
||||||
|
```
|
@ -103,16 +103,7 @@ This is a sample config for QuickSight:
|
|||||||
|
|
||||||
#### Source Configuration - Service Connection
|
#### Source Configuration - Service Connection
|
||||||
|
|
||||||
{% codeInfo srNumber=1 %}
|
{% partial file="/v1.6/connectors/yaml/common/aws-config-def.md" /%}
|
||||||
|
|
||||||
**awsConfig**
|
|
||||||
- **AWS Access Key ID**: Enter your secure access key ID for your Glue connection. The specified key ID should be authorized to read all databases you want to include in the metadata ingestion workflow.
|
|
||||||
- **AWS Secret Access Key**: Enter the Secret Access Key (the passcode key pair to the key ID from above).
|
|
||||||
- **AWS Region**: Enter the location of the amazon cluster that your data and account are associated with.
|
|
||||||
- **AWS Session Token (optional)**: The AWS session token is an optional parameter. If you want, enter the details of your temporary session token.
|
|
||||||
- **Endpoint URL (optional)**: Your Glue connector will automatically determine the AWS QuickSight endpoint URL based on the region. You may override this behavior by entering a value to the endpoint URL.
|
|
||||||
|
|
||||||
{% /codeInfo %}
|
|
||||||
|
|
||||||
{% codeInfo srNumber=2 %}
|
{% codeInfo srNumber=2 %}
|
||||||
|
|
||||||
@ -150,13 +141,9 @@ source:
|
|||||||
config:
|
config:
|
||||||
type: QuickSight
|
type: QuickSight
|
||||||
```
|
```
|
||||||
```yaml {% srNumber=1 %}
|
|
||||||
awsConfig:
|
{% partial file="/v1.6/connectors/yaml/common/aws-config.md" /%}
|
||||||
awsAccessKeyId: KEY
|
|
||||||
awsSecretAccessKey: SECRET
|
|
||||||
awsRegion: us-east-2
|
|
||||||
awsSessionToken: Token
|
|
||||||
```
|
|
||||||
```yaml {% srNumber=2 %}
|
```yaml {% srNumber=2 %}
|
||||||
awsAccountId: <aws-account-id>
|
awsAccountId: <aws-account-id>
|
||||||
```
|
```
|
||||||
|
@ -172,99 +172,7 @@ This is a sample config for Athena:
|
|||||||
|
|
||||||
#### Source Configuration - Service Connection
|
#### Source Configuration - Service Connection
|
||||||
|
|
||||||
{% codeInfo srNumber=1 %}
|
{% partial file="/v1.6/connectors/yaml/common/aws-config-def.md" /%}
|
||||||
|
|
||||||
- **awsAccessKeyId** & **awsSecretAccessKey**: When you interact with AWS, you specify your AWS security credentials to verify who you are and whether you have
|
|
||||||
permission to access the resources that you are requesting. AWS uses the security credentials to authenticate and
|
|
||||||
authorize your requests ([docs](https://docs.aws.amazon.com/IAM/latest/UserGuide/security-creds.html)).
|
|
||||||
|
|
||||||
Access keys consist of two parts: An **access key ID** (for example, `AKIAIOSFODNN7EXAMPLE`), and a **secret access key** (for example, `wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY`).
|
|
||||||
|
|
||||||
You must use both the access key ID and secret access key together to authenticate your requests.
|
|
||||||
|
|
||||||
You can find further information on how to manage your access keys [here](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html).
|
|
||||||
|
|
||||||
{% /codeInfo %}
|
|
||||||
|
|
||||||
{% codeInfo srNumber=2 %}
|
|
||||||
**awsSessionToken**: If you are using temporary credentials to access your services, you will need to inform the AWS Access Key ID
|
|
||||||
and AWS Secrets Access Key. Also, these will include an AWS Session Token.
|
|
||||||
|
|
||||||
{% /codeInfo %}
|
|
||||||
|
|
||||||
{% codeInfo srNumber=3 %}
|
|
||||||
|
|
||||||
**awsRegion**: Each AWS Region is a separate geographic area in which AWS clusters data centers ([docs](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.RegionsAndAvailabilityZones.html)).
|
|
||||||
|
|
||||||
As AWS can have instances in multiple regions, we need to know the region the service you want reach belongs to.
|
|
||||||
|
|
||||||
Note that the AWS Region is the only required parameter when configuring a connection. When connecting to the
|
|
||||||
services programmatically, there are different ways in which we can extract and use the rest of AWS configurations.
|
|
||||||
|
|
||||||
You can find further information about configuring your credentials [here](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html#configuring-credentials).
|
|
||||||
|
|
||||||
{% /codeInfo %}
|
|
||||||
|
|
||||||
{% codeInfo srNumber=4 %}
|
|
||||||
|
|
||||||
**endPointURL**: To connect programmatically to an AWS service, you use an endpoint. An *endpoint* is the URL of the
|
|
||||||
entry point for an AWS web service. The AWS SDKs and the AWS Command Line Interface (AWS CLI) automatically use the
|
|
||||||
default endpoint for each service in an AWS Region. But you can specify an alternate endpoint for your API requests.
|
|
||||||
|
|
||||||
Find more information on [AWS service endpoints](https://docs.aws.amazon.com/general/latest/gr/rande.html).
|
|
||||||
|
|
||||||
{% /codeInfo %}
|
|
||||||
|
|
||||||
{% codeInfo srNumber=5 %}
|
|
||||||
|
|
||||||
**profileName**: A named profile is a collection of settings and credentials that you can apply to a AWS CLI command.
|
|
||||||
When you specify a profile to run a command, the settings and credentials are used to run that command.
|
|
||||||
Multiple named profiles can be stored in the config and credentials files.
|
|
||||||
|
|
||||||
You can inform this field if you'd like to use a profile other than `default`.
|
|
||||||
|
|
||||||
Find here more information about [Named profiles for the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-profiles.html).
|
|
||||||
|
|
||||||
{% /codeInfo %}
|
|
||||||
|
|
||||||
{% codeInfo srNumber=6 %}
|
|
||||||
|
|
||||||
**assumeRoleArn**: Typically, you use `AssumeRole` within your account or for cross-account access. In this field you'll set the
|
|
||||||
`ARN` (Amazon Resource Name) of the policy of the other account.
|
|
||||||
|
|
||||||
A user who wants to access a role in a different account must also have permissions that are delegated from the account
|
|
||||||
administrator. The administrator must attach a policy that allows the user to call `AssumeRole` for the `ARN` of the role in the other account.
|
|
||||||
|
|
||||||
This is a required field if you'd like to `AssumeRole`.
|
|
||||||
|
|
||||||
Find more information on [AssumeRole](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html).
|
|
||||||
{% /codeInfo %}
|
|
||||||
|
|
||||||
{%note%}
|
|
||||||
When using Assume Role authentication, ensure you provide the following details:
|
|
||||||
- **AWS Region**: Specify the AWS region for your deployment.
|
|
||||||
- **Assume Role ARN**: Provide the ARN of the role in your AWS account that OpenMetadata will assume.
|
|
||||||
{%/note%}
|
|
||||||
|
|
||||||
{% codeInfo srNumber=7 %}
|
|
||||||
|
|
||||||
**assumeRoleSessionName**: An identifier for the assumed role session. Use the role session name to uniquely identify a session when the same role
|
|
||||||
is assumed by different principals or for different reasons.
|
|
||||||
|
|
||||||
By default, we'll use the name `OpenMetadataSession`.
|
|
||||||
|
|
||||||
Find more information about the [Role Session Name](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html#:~:text=An%20identifier%20for%20the%20assumed%20role%20session.).
|
|
||||||
|
|
||||||
{% /codeInfo %}
|
|
||||||
|
|
||||||
{% codeInfo srNumber=8 %}
|
|
||||||
|
|
||||||
**assumeRoleSourceIdentity**: The source identity specified by the principal that is calling the `AssumeRole` operation. You can use source identity
|
|
||||||
information in AWS CloudTrail logs to determine who took actions with a role.
|
|
||||||
|
|
||||||
Find more information about [Source Identity](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html#:~:text=Required%3A%20No-,SourceIdentity,-The%20source%20identity).
|
|
||||||
|
|
||||||
{% /codeInfo %}
|
|
||||||
|
|
||||||
{% codeInfo srNumber=9 %}
|
{% codeInfo srNumber=9 %}
|
||||||
|
|
||||||
@ -311,31 +219,9 @@ source:
|
|||||||
type: Athena
|
type: Athena
|
||||||
awsConfig:
|
awsConfig:
|
||||||
```
|
```
|
||||||
```yaml {% srNumber=1 %}
|
|
||||||
awsAccessKeyId: KEY
|
{% partial file="/v1.6/connectors/yaml/common/aws-config.md" /%}
|
||||||
awsSecretAccessKey: SECRET
|
|
||||||
```
|
|
||||||
```yaml {% srNumber=2 %}
|
|
||||||
# awsSessionToken: TOKEN
|
|
||||||
```
|
|
||||||
```yaml {% srNumber=3 %}
|
|
||||||
awsRegion: us-east-2
|
|
||||||
```
|
|
||||||
```yaml {% srNumber=4 %}
|
|
||||||
# endPointURL: https://athena.us-east-2.amazonaws.com/custom
|
|
||||||
```
|
|
||||||
```yaml {% srNumber=5 %}
|
|
||||||
# profileName: profile
|
|
||||||
```
|
|
||||||
```yaml {% srNumber=6 %}
|
|
||||||
# assumeRoleArn: "arn:partition:service:region:account:resource"
|
|
||||||
```
|
|
||||||
```yaml {% srNumber=7 %}
|
|
||||||
# assumeRoleSessionName: session
|
|
||||||
```
|
|
||||||
```yaml {% srNumber=8 %}
|
|
||||||
# assumeRoleSourceIdentity: identity
|
|
||||||
```
|
|
||||||
```yaml {% srNumber=9 %}
|
```yaml {% srNumber=9 %}
|
||||||
s3StagingDir: s3 directory for datasource
|
s3StagingDir: s3 directory for datasource
|
||||||
```
|
```
|
||||||
|
@ -78,35 +78,7 @@ This is a sample config for DynamoDB:
|
|||||||
|
|
||||||
#### Source Configuration - Service Connection
|
#### Source Configuration - Service Connection
|
||||||
|
|
||||||
{% codeInfo srNumber=1 %}
|
{% partial file="/v1.6/connectors/yaml/common/aws-config-def.md" /%}
|
||||||
|
|
||||||
**awsAccessKeyId**: Enter your secure access key ID for your DynamoDB connection. The specified key ID should be authorized to read all databases you want to include in the metadata ingestion workflow.
|
|
||||||
|
|
||||||
{% /codeInfo %}
|
|
||||||
|
|
||||||
{% codeInfo srNumber=2 %}
|
|
||||||
|
|
||||||
**awsSecretAccessKey**: Enter the Secret Access Key (the passcode key pair to the key ID from above).
|
|
||||||
|
|
||||||
{% /codeInfo %}
|
|
||||||
|
|
||||||
{% codeInfo srNumber=3 %}
|
|
||||||
|
|
||||||
**awsSessionToken**: The AWS session token is an optional parameter. If you want, enter the details of your temporary session token.
|
|
||||||
|
|
||||||
{% /codeInfo %}
|
|
||||||
|
|
||||||
{% codeInfo srNumber=4 %}
|
|
||||||
|
|
||||||
**awsRegion**: Enter the location of the amazon cluster that your data and account are associated with.
|
|
||||||
|
|
||||||
{% /codeInfo %}
|
|
||||||
|
|
||||||
{% codeInfo srNumber=5 %}
|
|
||||||
|
|
||||||
**endPointURL**: Your DynamoDB connector will automatically determine the AWS DynamoDB endpoint URL based on the region. You may override this behavior by entering a value to the endpoint URL.
|
|
||||||
|
|
||||||
{% /codeInfo %}
|
|
||||||
|
|
||||||
{% codeInfo srNumber=6 %}
|
{% codeInfo srNumber=6 %}
|
||||||
|
|
||||||
@ -149,21 +121,9 @@ source:
|
|||||||
type: DynamoDB
|
type: DynamoDB
|
||||||
awsConfig:
|
awsConfig:
|
||||||
```
|
```
|
||||||
```yaml {% srNumber=1 %}
|
|
||||||
awsAccessKeyId: aws_access_key_id
|
{% partial file="/v1.6/connectors/yaml/common/aws-config.md" /%}
|
||||||
```
|
|
||||||
```yaml {% srNumber=2 %}
|
|
||||||
awsSecretAccessKey: aws_secret_access_key
|
|
||||||
```
|
|
||||||
```yaml {% srNumber=3 %}
|
|
||||||
awsSessionToken: AWS Session Token
|
|
||||||
```
|
|
||||||
```yaml {% srNumber=4 %}
|
|
||||||
awsRegion: aws region
|
|
||||||
```
|
|
||||||
```yaml {% srNumber=5 %}
|
|
||||||
endPointURL: https://dynamodb.<region_name>.amazonaws.com
|
|
||||||
```
|
|
||||||
```yaml {% srNumber=6 %}
|
```yaml {% srNumber=6 %}
|
||||||
database: custom_database_name
|
database: custom_database_name
|
||||||
```
|
```
|
||||||
|
@ -58,35 +58,7 @@ This is a sample config for Glue:
|
|||||||
|
|
||||||
#### Source Configuration - Service Connection
|
#### Source Configuration - Service Connection
|
||||||
|
|
||||||
{% codeInfo srNumber=1 %}
|
{% partial file="/v1.6/connectors/yaml/common/aws-config-def.md" /%}
|
||||||
|
|
||||||
**awsAccessKeyId**: Enter your secure access key ID for your Glue connection. The specified key ID should be authorized to read all databases you want to include in the metadata ingestion workflow.
|
|
||||||
|
|
||||||
{% /codeInfo %}
|
|
||||||
|
|
||||||
{% codeInfo srNumber=2 %}
|
|
||||||
|
|
||||||
**awsSecretAccessKey**: Enter the Secret Access Key (the passcode key pair to the key ID from above).
|
|
||||||
|
|
||||||
{% /codeInfo %}
|
|
||||||
|
|
||||||
{% codeInfo srNumber=3 %}
|
|
||||||
|
|
||||||
**awsRegion**: Enter the location of the amazon cluster that your data and account are associated with.
|
|
||||||
|
|
||||||
{% /codeInfo %}
|
|
||||||
|
|
||||||
{% codeInfo srNumber=4 %}
|
|
||||||
|
|
||||||
**endPointURL**: Your Glue connector will automatically determine the AWS Glue endpoint URL based on the region. You may override this behavior by entering a value to the endpoint URL.
|
|
||||||
|
|
||||||
{% /codeInfo %}
|
|
||||||
|
|
||||||
{% codeInfo srNumber=5 %}
|
|
||||||
|
|
||||||
**awsSessionToken**: The AWS session token is an optional parameter. If you want, enter the details of your temporary session token.
|
|
||||||
|
|
||||||
{% /codeInfo %}
|
|
||||||
|
|
||||||
{% codeInfo srNumber=6 %}
|
{% codeInfo srNumber=6 %}
|
||||||
|
|
||||||
@ -130,21 +102,9 @@ source:
|
|||||||
type: Glue
|
type: Glue
|
||||||
awsConfig:
|
awsConfig:
|
||||||
```
|
```
|
||||||
```yaml {% srNumber=1 %}
|
|
||||||
awsAccessKeyId: KEY
|
{% partial file="/v1.6/connectors/yaml/common/aws-config.md" /%}
|
||||||
```
|
|
||||||
```yaml {% srNumber=2 %}
|
|
||||||
awsSecretAccessKey: SECRET
|
|
||||||
```
|
|
||||||
```yaml {% srNumber=3 %}
|
|
||||||
awsRegion: us-east-2
|
|
||||||
```
|
|
||||||
```yaml {% srNumber=4 %}
|
|
||||||
# endPointURL: https://glue.us-east-2.amazonaws.com/
|
|
||||||
```
|
|
||||||
```yaml {% srNumber=5 %}
|
|
||||||
# awsSessionToken: TOKEN
|
|
||||||
```
|
|
||||||
```yaml {% srNumber=6 %}
|
```yaml {% srNumber=6 %}
|
||||||
databaseName: database_name
|
databaseName: database_name
|
||||||
```
|
```
|
||||||
|
@ -90,13 +90,11 @@ The workflow is modeled around the following JSON Schema.
|
|||||||
|
|
||||||
{% codeInfo srNumber=1 %}
|
{% codeInfo srNumber=1 %}
|
||||||
|
|
||||||
* **awsAccessKeyId**: Enter your secure access key ID for your DynamoDB connection. The specified key ID should be authorized to read all databases you want to include in the metadata ingestion workflow.
|
|
||||||
* **awsSecretAccessKey**: Enter the Secret Access Key (the passcode key pair to the key ID from above).
|
|
||||||
* **awsRegion**: Specify the region in which your DynamoDB is located. This setting is required even if you have configured a local AWS profile.
|
|
||||||
* **schemaFilterPattern** and **tableFilterPattern**: Note that the `schemaFilterPattern` and `tableFilterPattern` both support regex as `include` or `exclude`. E.g.,
|
* **schemaFilterPattern** and **tableFilterPattern**: Note that the `schemaFilterPattern` and `tableFilterPattern` both support regex as `include` or `exclude`. E.g.,
|
||||||
|
|
||||||
{% /codeInfo %}
|
{% /codeInfo %}
|
||||||
|
|
||||||
|
{% partial file="/v1.6/connectors/yaml/common/aws-config-def.md" /%}
|
||||||
|
|
||||||
{% partial file="/v1.6/connectors/yaml/database/source-config-def.md" /%}
|
{% partial file="/v1.6/connectors/yaml/database/source-config-def.md" /%}
|
||||||
|
|
||||||
@ -120,9 +118,11 @@ source:
|
|||||||
```yaml {% srNumber=1 %}
|
```yaml {% srNumber=1 %}
|
||||||
configSource:
|
configSource:
|
||||||
securityConfig:
|
securityConfig:
|
||||||
awsAccessKeyId: aws access key id
|
```
|
||||||
awsSecretAccessKey: aws secret access key
|
|
||||||
awsRegion: aws region
|
{% partial file="/v1.6/connectors/yaml/common/aws-config.md" /%}
|
||||||
|
|
||||||
|
```yaml {% srNumber=1 %}
|
||||||
bucketName: bucket name
|
bucketName: bucket name
|
||||||
prefix: prefix
|
prefix: prefix
|
||||||
```
|
```
|
||||||
|
@ -46,38 +46,7 @@ In this configuration we will be fetching the dbt `manifest.json`, `catalog.json
|
|||||||
- **dbtConfigType**: s3
|
- **dbtConfigType**: s3
|
||||||
{% /codeInfo %}
|
{% /codeInfo %}
|
||||||
|
|
||||||
{% codeInfo srNumber=1 %}
|
{% partial file="/v1.6/connectors/yaml/common/aws-config-def.md" /%}
|
||||||
|
|
||||||
- **awsAccessKeyId** & **awsSecretAccessKey**: When you interact with AWS, you specify your AWS security credentials to verify who you are and whether you have
|
|
||||||
permission to access the resources that you are requesting. AWS uses the security credentials to authenticate and
|
|
||||||
authorize your requests ([docs](https://docs.aws.amazon.com/IAM/latest/UserGuide/security-creds.html)).
|
|
||||||
|
|
||||||
Access keys consist of two parts: An **access key ID** (for example, `AKIAIOSFODNN7EXAMPLE`), and a **secret access key** (for example, `wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY`).
|
|
||||||
|
|
||||||
You must use both the access key ID and secret access key together to authenticate your requests.
|
|
||||||
|
|
||||||
You can find further information on how to manage your access keys [here](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html).
|
|
||||||
|
|
||||||
{% /codeInfo %}
|
|
||||||
|
|
||||||
{% codeInfo srNumber=2 %}
|
|
||||||
**awsSessionToken**: If you are using temporary credentials to access your services, you will need to inform the AWS Access Key ID
|
|
||||||
and AWS Secrets Access Key. Also, these will include an AWS Session Token.
|
|
||||||
|
|
||||||
{% /codeInfo %}
|
|
||||||
|
|
||||||
{% codeInfo srNumber=3 %}
|
|
||||||
|
|
||||||
**awsRegion**: Each AWS Region is a separate geographic area in which AWS clusters data centers ([docs](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.RegionsAndAvailabilityZones.html)).
|
|
||||||
|
|
||||||
As AWS can have instances in multiple regions, we need to know the region the service you want reach belongs to.
|
|
||||||
|
|
||||||
Note that the AWS Region is the only required parameter when configuring a connection. When connecting to the
|
|
||||||
services programmatically, there are different ways in which we can extract and use the rest of AWS configurations.
|
|
||||||
|
|
||||||
You can find further information about configuring your credentials [here](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html#configuring-credentials).
|
|
||||||
|
|
||||||
{% /codeInfo %}
|
|
||||||
|
|
||||||
{% partial file="/v1.6/connectors/yaml/dbt/dbt-prefix-def.md" /%}
|
{% partial file="/v1.6/connectors/yaml/dbt/dbt-prefix-def.md" /%}
|
||||||
|
|
||||||
@ -105,16 +74,10 @@ source:
|
|||||||
```
|
```
|
||||||
```yaml {% srNumber=1 %}
|
```yaml {% srNumber=1 %}
|
||||||
dbtSecurityConfig:
|
dbtSecurityConfig:
|
||||||
awsAccessKeyId: AWS_ACCESS_KEY_ID
|
|
||||||
awsSecretAccessKey: AWS_SECRET_ACCESS_KEY
|
|
||||||
```
|
|
||||||
```yaml {% srNumber=2 %}
|
|
||||||
awsSessionToken: AWS_SESSION_TOKEN
|
|
||||||
```
|
|
||||||
```yaml {% srNumber=3 %}
|
|
||||||
awsRegion: AWS_REGION
|
|
||||||
```
|
```
|
||||||
|
|
||||||
|
{% partial file="/v1.6/connectors/yaml/common/aws-config.md" /%}
|
||||||
|
|
||||||
{% partial file="/v1.6/connectors/yaml/dbt/dbt-prefix.md" /%}
|
{% partial file="/v1.6/connectors/yaml/dbt/dbt-prefix.md" /%}
|
||||||
|
|
||||||
{% partial file="/v1.6/connectors/yaml/dbt/source-config.md" /%}
|
{% partial file="/v1.6/connectors/yaml/dbt/source-config.md" /%}
|
||||||
|
@ -80,100 +80,7 @@ This is a sample config for Kinesis:
|
|||||||
|
|
||||||
#### Source Configuration - Service Connection
|
#### Source Configuration - Service Connection
|
||||||
|
|
||||||
{% codeInfo srNumber=1 %}
|
{% partial file="/v1.6/connectors/yaml/common/aws-config-def.md" /%}
|
||||||
|
|
||||||
- **awsAccessKeyId** & **awsSecretAccessKey**: When you interact with AWS, you specify your AWS security credentials to verify who you are and whether you have
|
|
||||||
permission to access the resources that you are requesting. AWS uses the security credentials to authenticate and
|
|
||||||
authorize your requests ([docs](https://docs.aws.amazon.com/IAM/latest/UserGuide/security-creds.html)).
|
|
||||||
|
|
||||||
Access keys consist of two parts: An **access key ID** (for example, `AKIAIOSFODNN7EXAMPLE`), and a **secret access key** (for example, `wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY`).
|
|
||||||
|
|
||||||
You must use both the access key ID and secret access key together to authenticate your requests.
|
|
||||||
|
|
||||||
You can find further information on how to manage your access keys [here](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html).
|
|
||||||
|
|
||||||
{% /codeInfo %}
|
|
||||||
|
|
||||||
{% codeInfo srNumber=2 %}
|
|
||||||
**awsSessionToken**: If you are using temporary credentials to access your services, you will need to inform the AWS Access Key ID
|
|
||||||
and AWS Secrets Access Key. Also, these will include an AWS Session Token.
|
|
||||||
|
|
||||||
{% /codeInfo %}
|
|
||||||
|
|
||||||
{% codeInfo srNumber=3 %}
|
|
||||||
|
|
||||||
**awsRegion**: Each AWS Region is a separate geographic area in which AWS clusters data centers ([docs](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.RegionsAndAvailabilityZones.html)).
|
|
||||||
|
|
||||||
As AWS can have instances in multiple regions, we need to know the region the service you want reach belongs to.
|
|
||||||
|
|
||||||
Note that the AWS Region is the only required parameter when configuring a connection. When connecting to the
|
|
||||||
services programmatically, there are different ways in which we can extract and use the rest of AWS configurations.
|
|
||||||
|
|
||||||
You can find further information about configuring your credentials [here](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html#configuring-credentials).
|
|
||||||
|
|
||||||
{% /codeInfo %}
|
|
||||||
|
|
||||||
{% codeInfo srNumber=4 %}
|
|
||||||
|
|
||||||
**endPointURL**: To connect programmatically to an AWS service, you use an endpoint. An *endpoint* is the URL of the
|
|
||||||
entry point for an AWS web service. The AWS SDKs and the AWS Command Line Interface (AWS CLI) automatically use the
|
|
||||||
default endpoint for each service in an AWS Region. But you can specify an alternate endpoint for your API requests.
|
|
||||||
|
|
||||||
Find more information on [AWS service endpoints](https://docs.aws.amazon.com/general/latest/gr/rande.html).
|
|
||||||
|
|
||||||
{% /codeInfo %}
|
|
||||||
|
|
||||||
{% codeInfo srNumber=5 %}
|
|
||||||
|
|
||||||
**profileName**: A named profile is a collection of settings and credentials that you can apply to a AWS CLI command.
|
|
||||||
When you specify a profile to run a command, the settings and credentials are used to run that command.
|
|
||||||
Multiple named profiles can be stored in the config and credentials files.
|
|
||||||
|
|
||||||
You can inform this field if you'd like to use a profile other than `default`.
|
|
||||||
|
|
||||||
Find here more information about [Named profiles for the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-profiles.html).
|
|
||||||
|
|
||||||
{% /codeInfo %}
|
|
||||||
|
|
||||||
{% codeInfo srNumber=6 %}
|
|
||||||
|
|
||||||
**assumeRoleArn**: Typically, you use `AssumeRole` within your account or for cross-account access. In this field you'll set the
|
|
||||||
`ARN` (Amazon Resource Name) of the policy of the other account.
|
|
||||||
|
|
||||||
A user who wants to access a role in a different account must also have permissions that are delegated from the account
|
|
||||||
administrator. The administrator must attach a policy that allows the user to call `AssumeRole` for the `ARN` of the role in the other account.
|
|
||||||
|
|
||||||
This is a required field if you'd like to `AssumeRole`.
|
|
||||||
|
|
||||||
Find more information on [AssumeRole](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html).
|
|
||||||
|
|
||||||
{%note%}
|
|
||||||
When using Assume Role authentication, ensure you provide the following details:
|
|
||||||
- **AWS Region**: Specify the AWS region for your deployment.
|
|
||||||
- **Assume Role ARN**: Provide the ARN of the role in your AWS account that OpenMetadata will assume.
|
|
||||||
{%/note%}
|
|
||||||
|
|
||||||
{% /codeInfo %}
|
|
||||||
|
|
||||||
{% codeInfo srNumber=7 %}
|
|
||||||
|
|
||||||
**assumeRoleSessionName**: An identifier for the assumed role session. Use the role session name to uniquely identify a session when the same role
|
|
||||||
is assumed by different principals or for different reasons.
|
|
||||||
|
|
||||||
By default, we'll use the name `OpenMetadataSession`.
|
|
||||||
|
|
||||||
Find more information about the [Role Session Name](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html#:~:text=An%20identifier%20for%20the%20assumed%20role%20session.).
|
|
||||||
|
|
||||||
{% /codeInfo %}
|
|
||||||
|
|
||||||
{% codeInfo srNumber=8 %}
|
|
||||||
|
|
||||||
**assumeRoleSourceIdentity**: The source identity specified by the principal that is calling the `AssumeRole` operation. You can use source identity
|
|
||||||
information in AWS CloudTrail logs to determine who took actions with a role.
|
|
||||||
|
|
||||||
Find more information about [Source Identity](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html#:~:text=Required%3A%20No-,SourceIdentity,-The%20source%20identity).
|
|
||||||
|
|
||||||
{% /codeInfo %}
|
|
||||||
|
|
||||||
{% partial file="/v1.6/connectors/yaml/messaging/source-config-def.md" /%}
|
{% partial file="/v1.6/connectors/yaml/messaging/source-config-def.md" /%}
|
||||||
|
|
||||||
@ -195,31 +102,9 @@ source:
|
|||||||
type: Kinesis
|
type: Kinesis
|
||||||
awsConfig:
|
awsConfig:
|
||||||
```
|
```
|
||||||
```yaml {% srNumber=1 %}
|
|
||||||
awsAccessKeyId: KEY
|
{% partial file="/v1.6/connectors/yaml/common/aws-config.md" /%}
|
||||||
awsSecretAccessKey: SECRET
|
|
||||||
```
|
|
||||||
```yaml {% srNumber=2 %}
|
|
||||||
# awsSessionToken: TOKEN
|
|
||||||
```
|
|
||||||
```yaml {% srNumber=3 %}
|
|
||||||
awsRegion: us-east-2
|
|
||||||
```
|
|
||||||
```yaml {% srNumber=4 %}
|
|
||||||
# endPointURL: https://athena.us-east-2.amazonaws.com/custom
|
|
||||||
```
|
|
||||||
```yaml {% srNumber=5 %}
|
|
||||||
# profileName: profile
|
|
||||||
```
|
|
||||||
```yaml {% srNumber=6 %}
|
|
||||||
# assumeRoleArn: "arn:partition:service:region:account:resource"
|
|
||||||
```
|
|
||||||
```yaml {% srNumber=7 %}
|
|
||||||
# assumeRoleSessionName: session
|
|
||||||
```
|
|
||||||
```yaml {% srNumber=8 %}
|
|
||||||
# assumeRoleSourceIdentity: identity
|
|
||||||
```
|
|
||||||
|
|
||||||
{% partial file="/v1.6/connectors/yaml/messaging/source-config.md" /%}
|
{% partial file="/v1.6/connectors/yaml/messaging/source-config.md" /%}
|
||||||
|
|
||||||
|
@ -78,100 +78,7 @@ This is a sample config for Sagemaker:
|
|||||||
|
|
||||||
#### Source Configuration - Service Connection
|
#### Source Configuration - Service Connection
|
||||||
|
|
||||||
{% codeInfo srNumber=1 %}
|
{% partial file="/v1.6/connectors/yaml/common/aws-config-def.md" /%}
|
||||||
|
|
||||||
- **awsAccessKeyId** & **awsSecretAccessKey**: When you interact with AWS, you specify your AWS security credentials to verify who you are and whether you have
|
|
||||||
permission to access the resources that you are requesting. AWS uses the security credentials to authenticate and
|
|
||||||
authorize your requests ([docs](https://docs.aws.amazon.com/IAM/latest/UserGuide/security-creds.html)).
|
|
||||||
|
|
||||||
Access keys consist of two parts: An **access key ID** (for example, `AKIAIOSFODNN7EXAMPLE`), and a **secret access key** (for example, `wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY`).
|
|
||||||
|
|
||||||
You must use both the access key ID and secret access key together to authenticate your requests.
|
|
||||||
|
|
||||||
You can find further information on how to manage your access keys [here](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html).
|
|
||||||
|
|
||||||
{% /codeInfo %}
|
|
||||||
|
|
||||||
{% codeInfo srNumber=2 %}
|
|
||||||
**awsSessionToken**: If you are using temporary credentials to access your services, you will need to inform the AWS Access Key ID
|
|
||||||
and AWS Secrets Access Key. Also, these will include an AWS Session Token.
|
|
||||||
|
|
||||||
{% /codeInfo %}
|
|
||||||
|
|
||||||
{% codeInfo srNumber=3 %}
|
|
||||||
|
|
||||||
**awsRegion**: Each AWS Region is a separate geographic area in which AWS clusters data centers ([docs](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.RegionsAndAvailabilityZones.html)).
|
|
||||||
|
|
||||||
As AWS can have instances in multiple regions, we need to know the region the service you want reach belongs to.
|
|
||||||
|
|
||||||
Note that the AWS Region is the only required parameter when configuring a connection. When connecting to the
|
|
||||||
services programmatically, there are different ways in which we can extract and use the rest of AWS configurations.
|
|
||||||
|
|
||||||
You can find further information about configuring your credentials [here](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html#configuring-credentials).
|
|
||||||
|
|
||||||
{% /codeInfo %}
|
|
||||||
|
|
||||||
{% codeInfo srNumber=4 %}
|
|
||||||
|
|
||||||
**endPointURL**: To connect programmatically to an AWS service, you use an endpoint. An *endpoint* is the URL of the
|
|
||||||
entry point for an AWS web service. The AWS SDKs and the AWS Command Line Interface (AWS CLI) automatically use the
|
|
||||||
default endpoint for each service in an AWS Region. But you can specify an alternate endpoint for your API requests.
|
|
||||||
|
|
||||||
Find more information on [AWS service endpoints](https://docs.aws.amazon.com/general/latest/gr/rande.html).
|
|
||||||
|
|
||||||
{% /codeInfo %}
|
|
||||||
|
|
||||||
{% codeInfo srNumber=5 %}
|
|
||||||
|
|
||||||
**profileName**: A named profile is a collection of settings and credentials that you can apply to a AWS CLI command.
|
|
||||||
When you specify a profile to run a command, the settings and credentials are used to run that command.
|
|
||||||
Multiple named profiles can be stored in the config and credentials files.
|
|
||||||
|
|
||||||
You can inform this field if you'd like to use a profile other than `default`.
|
|
||||||
|
|
||||||
Find here more information about [Named profiles for the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-profiles.html).
|
|
||||||
|
|
||||||
{% /codeInfo %}
|
|
||||||
|
|
||||||
{% codeInfo srNumber=6 %}
|
|
||||||
|
|
||||||
**assumeRoleArn**: Typically, you use `AssumeRole` within your account or for cross-account access. In this field you'll set the
|
|
||||||
`ARN` (Amazon Resource Name) of the policy of the other account.
|
|
||||||
|
|
||||||
A user who wants to access a role in a different account must also have permissions that are delegated from the account
|
|
||||||
administrator. The administrator must attach a policy that allows the user to call `AssumeRole` for the `ARN` of the role in the other account.
|
|
||||||
|
|
||||||
This is a required field if you'd like to `AssumeRole`.
|
|
||||||
|
|
||||||
Find more information on [AssumeRole](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html).
|
|
||||||
|
|
||||||
{%note%}
|
|
||||||
When using Assume Role authentication, ensure you provide the following details:
|
|
||||||
- **AWS Region**: Specify the AWS region for your deployment.
|
|
||||||
- **Assume Role ARN**: Provide the ARN of the role in your AWS account that OpenMetadata will assume.
|
|
||||||
{%/note%}
|
|
||||||
|
|
||||||
{% /codeInfo %}
|
|
||||||
|
|
||||||
{% codeInfo srNumber=7 %}
|
|
||||||
|
|
||||||
**assumeRoleSessionName**: An identifier for the assumed role session. Use the role session name to uniquely identify a session when the same role
|
|
||||||
is assumed by different principals or for different reasons.
|
|
||||||
|
|
||||||
By default, we'll use the name `OpenMetadataSession`.
|
|
||||||
|
|
||||||
Find more information about the [Role Session Name](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html#:~:text=An%20identifier%20for%20the%20assumed%20role%20session.).
|
|
||||||
|
|
||||||
{% /codeInfo %}
|
|
||||||
|
|
||||||
{% codeInfo srNumber=8 %}
|
|
||||||
|
|
||||||
**assumeRoleSourceIdentity**: The source identity specified by the principal that is calling the `AssumeRole` operation. You can use source identity
|
|
||||||
information in AWS CloudTrail logs to determine who took actions with a role.
|
|
||||||
|
|
||||||
Find more information about [Source Identity](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html#:~:text=Required%3A%20No-,SourceIdentity,-The%20source%20identity).
|
|
||||||
|
|
||||||
{% /codeInfo %}
|
|
||||||
|
|
||||||
{% partial file="/v1.6/connectors/yaml/ml-model/source-config-def.md" /%}
|
{% partial file="/v1.6/connectors/yaml/ml-model/source-config-def.md" /%}
|
||||||
|
|
||||||
@ -193,31 +100,8 @@ source:
|
|||||||
type: SageMaker
|
type: SageMaker
|
||||||
awsConfig:
|
awsConfig:
|
||||||
```
|
```
|
||||||
```yaml {% srNumber=1 %}
|
|
||||||
awsAccessKeyId: KEY
|
{% partial file="/v1.6/connectors/yaml/common/aws-config.md" /%}
|
||||||
awsSecretAccessKey: SECRET
|
|
||||||
```
|
|
||||||
```yaml {% srNumber=2 %}
|
|
||||||
# awsSessionToken: TOKEN
|
|
||||||
```
|
|
||||||
```yaml {% srNumber=3 %}
|
|
||||||
awsRegion: us-east-2
|
|
||||||
```
|
|
||||||
```yaml {% srNumber=4 %}
|
|
||||||
# endPointURL: https://athena.us-east-2.amazonaws.com/custom
|
|
||||||
```
|
|
||||||
```yaml {% srNumber=5 %}
|
|
||||||
# profileName: profile
|
|
||||||
```
|
|
||||||
```yaml {% srNumber=6 %}
|
|
||||||
# assumeRoleArn: "arn:partition:service:region:account:resource"
|
|
||||||
```
|
|
||||||
```yaml {% srNumber=7 %}
|
|
||||||
# assumeRoleSessionName: session
|
|
||||||
```
|
|
||||||
```yaml {% srNumber=8 %}
|
|
||||||
# assumeRoleSourceIdentity: identity
|
|
||||||
```
|
|
||||||
|
|
||||||
{% partial file="/v1.6/connectors/yaml/ml-model/source-config.md" /%}
|
{% partial file="/v1.6/connectors/yaml/ml-model/source-config.md" /%}
|
||||||
|
|
||||||
|
@ -63,58 +63,7 @@ This is a sample config for Glue:
|
|||||||
|
|
||||||
#### Source Configuration - Service Connection
|
#### Source Configuration - Service Connection
|
||||||
|
|
||||||
{% codeInfo srNumber=1 %}
|
{% partial file="/v1.6/connectors/yaml/common/aws-config-def.md" /%}
|
||||||
|
|
||||||
|
|
||||||
**awsAccessKeyId**: Enter your secure access key ID for your Glue connection. The specified key ID should be
|
|
||||||
authorized to read all databases you want to include in the metadata ingestion workflow.
|
|
||||||
|
|
||||||
{% /codeInfo %}
|
|
||||||
|
|
||||||
{% codeInfo srNumber=2 %}
|
|
||||||
|
|
||||||
**awsSecretAccessKey**: Enter the Secret Access Key (the passcode key pair to the key ID from above).
|
|
||||||
|
|
||||||
|
|
||||||
{% /codeInfo %}
|
|
||||||
|
|
||||||
{% codeInfo srNumber=3 %}
|
|
||||||
|
|
||||||
**awsRegion**: Enter the location of the amazon cluster that your data and account are associated with.
|
|
||||||
|
|
||||||
{% /codeInfo %}
|
|
||||||
|
|
||||||
{% codeInfo srNumber=4 %}
|
|
||||||
|
|
||||||
**awsSessionToken**: The AWS session token is an optional parameter. If you want, enter the details of your temporary
|
|
||||||
session token.
|
|
||||||
|
|
||||||
{% /codeInfo %}
|
|
||||||
|
|
||||||
{% codeInfo srNumber=5 %}
|
|
||||||
|
|
||||||
**endPointURL**: Your Glue connector will automatically determine the AWS Glue endpoint URL based on the region. You
|
|
||||||
may override this behavior by entering a value to the endpoint URL.
|
|
||||||
|
|
||||||
|
|
||||||
{% /codeInfo %}
|
|
||||||
|
|
||||||
{% codeInfo srNumber=6 %}
|
|
||||||
**profileName**: The name of a profile to use with the boto session.
|
|
||||||
{% /codeInfo %}
|
|
||||||
|
|
||||||
{% codeInfo srNumber=7 %}
|
|
||||||
**assumeRoleArn**: The Amazon Resource Name (ARN) of the role to assume. Required Field in case of Assume Role.
|
|
||||||
{% /codeInfo %}
|
|
||||||
|
|
||||||
{% codeInfo srNumber=8 %}
|
|
||||||
**assumeRoleSessionName**: An identifier for the assumed role session. Use the role session name to uniquely identify a session when the same role is assumed by different principals or for different reasons. Required Field in case of Assume Role.
|
|
||||||
{% /codeInfo %}
|
|
||||||
|
|
||||||
{% codeInfo srNumber=9 %}
|
|
||||||
**assumeRoleSourceIdentity**: The Amazon Resource Name (ARN) of the role to assume. Optional Field in case of Assume Role.
|
|
||||||
{% /codeInfo %}
|
|
||||||
|
|
||||||
|
|
||||||
{% partial file="/v1.6/connectors/yaml/pipeline/source-config-def.md" /%}
|
{% partial file="/v1.6/connectors/yaml/pipeline/source-config-def.md" /%}
|
||||||
|
|
||||||
@ -135,34 +84,8 @@ source:
|
|||||||
type: GluePipeline
|
type: GluePipeline
|
||||||
awsConfig:
|
awsConfig:
|
||||||
```
|
```
|
||||||
```yaml {% srNumber=1 %}
|
|
||||||
awsAccessKeyId: KEY
|
|
||||||
```
|
|
||||||
```yaml {% srNumber=2 %}
|
|
||||||
awsSecretAccessKey: SECRET
|
|
||||||
```
|
|
||||||
```yaml {% srNumber=3 %}
|
|
||||||
awsRegion: us-east-2
|
|
||||||
```
|
|
||||||
```yaml {% srNumber=4 %}
|
|
||||||
# awsSessionToken: TOKEN
|
|
||||||
```
|
|
||||||
```yaml {% srNumber=5 %}
|
|
||||||
# endPointURL: https://glue.us-east-2.amazonaws.com/
|
|
||||||
```
|
|
||||||
```yaml {% srNumber=6 %}
|
|
||||||
# profileName: ""
|
|
||||||
```
|
|
||||||
```yaml {% srNumber=7 %}
|
|
||||||
# assumeRoleArn: ""
|
|
||||||
```
|
|
||||||
```yaml {% srNumber=8 %}
|
|
||||||
# assumeRoleSessionName: OpenMetadataSession
|
|
||||||
```
|
|
||||||
```yaml {% srNumber=9 %}
|
|
||||||
# assumeRoleSourceIdentity : ""
|
|
||||||
```
|
|
||||||
|
|
||||||
|
{% partial file="/v1.6/connectors/yaml/common/aws-config.md" /%}
|
||||||
|
|
||||||
{% partial file="/v1.6/connectors/yaml/pipeline/source-config.md" /%}
|
{% partial file="/v1.6/connectors/yaml/pipeline/source-config.md" /%}
|
||||||
|
|
||||||
|
@ -128,100 +128,7 @@ This is a sample config for Athena:
|
|||||||
|
|
||||||
#### Source Configuration - Service Connection
|
#### Source Configuration - Service Connection
|
||||||
|
|
||||||
{% codeInfo srNumber=1 %}
|
{% partial file="/v1.6/connectors/yaml/common/aws-config-def.md" /%}
|
||||||
|
|
||||||
- **awsAccessKeyId** & **awsSecretAccessKey**: When you interact with AWS, you specify your AWS security credentials to verify who you are and whether you have
|
|
||||||
permission to access the resources that you are requesting. AWS uses the security credentials to authenticate and
|
|
||||||
authorize your requests ([docs](https://docs.aws.amazon.com/IAM/latest/UserGuide/security-creds.html)).
|
|
||||||
|
|
||||||
Access keys consist of two parts: An **access key ID** (for example, `AKIAIOSFODNN7EXAMPLE`), and a **secret access key** (for example, `wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY`).
|
|
||||||
|
|
||||||
You must use both the access key ID and secret access key together to authenticate your requests.
|
|
||||||
|
|
||||||
You can find further information on how to manage your access keys [here](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html).
|
|
||||||
|
|
||||||
{% /codeInfo %}
|
|
||||||
|
|
||||||
{% codeInfo srNumber=2 %}
|
|
||||||
**awsSessionToken**: If you are using temporary credentials to access your services, you will need to inform the AWS Access Key ID
|
|
||||||
and AWS Secrets Access Key. Also, these will include an AWS Session Token.
|
|
||||||
|
|
||||||
{% /codeInfo %}
|
|
||||||
|
|
||||||
{% codeInfo srNumber=3 %}
|
|
||||||
|
|
||||||
**awsRegion**: Each AWS Region is a separate geographic area in which AWS clusters data centers ([docs](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.RegionsAndAvailabilityZones.html)).
|
|
||||||
|
|
||||||
As AWS can have instances in multiple regions, we need to know the region the service you want reach belongs to.
|
|
||||||
|
|
||||||
Note that the AWS Region is the only required parameter when configuring a connection. When connecting to the
|
|
||||||
services programmatically, there are different ways in which we can extract and use the rest of AWS configurations.
|
|
||||||
|
|
||||||
You can find further information about configuring your credentials [here](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html#configuring-credentials).
|
|
||||||
|
|
||||||
{% /codeInfo %}
|
|
||||||
|
|
||||||
{% codeInfo srNumber=4 %}
|
|
||||||
|
|
||||||
**endPointURL**: To connect programmatically to an AWS service, you use an endpoint. An *endpoint* is the URL of the
|
|
||||||
entry point for an AWS web service. The AWS SDKs and the AWS Command Line Interface (AWS CLI) automatically use the
|
|
||||||
default endpoint for each service in an AWS Region. But you can specify an alternate endpoint for your API requests.
|
|
||||||
|
|
||||||
Find more information on [AWS service endpoints](https://docs.aws.amazon.com/general/latest/gr/rande.html).
|
|
||||||
|
|
||||||
{% /codeInfo %}
|
|
||||||
|
|
||||||
{% codeInfo srNumber=5 %}
|
|
||||||
|
|
||||||
**profileName**: A named profile is a collection of settings and credentials that you can apply to a AWS CLI command.
|
|
||||||
When you specify a profile to run a command, the settings and credentials are used to run that command.
|
|
||||||
Multiple named profiles can be stored in the config and credentials files.
|
|
||||||
|
|
||||||
You can inform this field if you'd like to use a profile other than `default`.
|
|
||||||
|
|
||||||
Find here more information about [Named profiles for the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-profiles.html).
|
|
||||||
|
|
||||||
{% /codeInfo %}
|
|
||||||
|
|
||||||
{% codeInfo srNumber=6 %}
|
|
||||||
|
|
||||||
**assumeRoleArn**: Typically, you use `AssumeRole` within your account or for cross-account access. In this field you'll set the
|
|
||||||
`ARN` (Amazon Resource Name) of the policy of the other account.
|
|
||||||
|
|
||||||
A user who wants to access a role in a different account must also have permissions that are delegated from the account
|
|
||||||
administrator. The administrator must attach a policy that allows the user to call `AssumeRole` for the `ARN` of the role in the other account.
|
|
||||||
|
|
||||||
This is a required field if you'd like to `AssumeRole`.
|
|
||||||
|
|
||||||
Find more information on [AssumeRole](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html).
|
|
||||||
|
|
||||||
{%note%}
|
|
||||||
When using Assume Role authentication, ensure you provide the following details:
|
|
||||||
- **AWS Region**: Specify the AWS region for your deployment.
|
|
||||||
- **Assume Role ARN**: Provide the ARN of the role in your AWS account that OpenMetadata will assume.
|
|
||||||
{%/note%}
|
|
||||||
|
|
||||||
{% /codeInfo %}
|
|
||||||
|
|
||||||
{% codeInfo srNumber=7 %}
|
|
||||||
|
|
||||||
**assumeRoleSessionName**: An identifier for the assumed role session. Use the role session name to uniquely identify a session when the same role
|
|
||||||
is assumed by different principals or for different reasons.
|
|
||||||
|
|
||||||
By default, we'll use the name `OpenMetadataSession`.
|
|
||||||
|
|
||||||
Find more information about the [Role Session Name](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html#:~:text=An%20identifier%20for%20the%20assumed%20role%20session.).
|
|
||||||
|
|
||||||
{% /codeInfo %}
|
|
||||||
|
|
||||||
{% codeInfo srNumber=8 %}
|
|
||||||
|
|
||||||
**assumeRoleSourceIdentity**: The source identity specified by the principal that is calling the `AssumeRole` operation. You can use source identity
|
|
||||||
information in AWS CloudTrail logs to determine who took actions with a role.
|
|
||||||
|
|
||||||
Find more information about [Source Identity](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html#:~:text=Required%3A%20No-,SourceIdentity,-The%20source%20identity).
|
|
||||||
|
|
||||||
{% /codeInfo %}
|
|
||||||
|
|
||||||
{% partial file="/v1.6/connectors/yaml/storage/source-config-def.md" /%}
|
{% partial file="/v1.6/connectors/yaml/storage/source-config-def.md" /%}
|
||||||
|
|
||||||
@ -262,31 +169,9 @@ source:
|
|||||||
type: S3
|
type: S3
|
||||||
awsConfig:
|
awsConfig:
|
||||||
```
|
```
|
||||||
```yaml {% srNumber=1 %}
|
|
||||||
awsAccessKeyId: KEY
|
{% partial file="/v1.6/connectors/yaml/common/aws-config.md" /%}
|
||||||
awsSecretAccessKey: SECRET
|
|
||||||
```
|
|
||||||
```yaml {% srNumber=2 %}
|
|
||||||
# awsSessionToken: TOKEN
|
|
||||||
```
|
|
||||||
```yaml {% srNumber=3 %}
|
|
||||||
awsRegion: us-east-2
|
|
||||||
```
|
|
||||||
```yaml {% srNumber=4 %}
|
|
||||||
# endPointURL: https://s3.us-east-2.amazonaws.com/custom
|
|
||||||
```
|
|
||||||
```yaml {% srNumber=5 %}
|
|
||||||
# profileName: profile
|
|
||||||
```
|
|
||||||
```yaml {% srNumber=6 %}
|
|
||||||
# assumeRoleArn: "arn:partition:service:region:account:resource"
|
|
||||||
```
|
|
||||||
```yaml {% srNumber=7 %}
|
|
||||||
# assumeRoleSessionName: session
|
|
||||||
```
|
|
||||||
```yaml {% srNumber=8 %}
|
|
||||||
# assumeRoleSourceIdentity: identity
|
|
||||||
```
|
|
||||||
```yaml {% srNumber=13 %}
|
```yaml {% srNumber=13 %}
|
||||||
bucketNames:
|
bucketNames:
|
||||||
- s3-testing-1
|
- s3-testing-1
|
||||||
|
@ -103,16 +103,7 @@ This is a sample config for QuickSight:
|
|||||||
|
|
||||||
#### Source Configuration - Service Connection
|
#### Source Configuration - Service Connection
|
||||||
|
|
||||||
{% codeInfo srNumber=1 %}
|
{% partial file="/v1.7/connectors/yaml/common/aws-config-def.md" /%}
|
||||||
|
|
||||||
**awsConfig**
|
|
||||||
- **AWS Access Key ID**: Enter your secure access key ID for your Glue connection. The specified key ID should be authorized to read all databases you want to include in the metadata ingestion workflow.
|
|
||||||
- **AWS Secret Access Key**: Enter the Secret Access Key (the passcode key pair to the key ID from above).
|
|
||||||
- **AWS Region**: Enter the location of the amazon cluster that your data and account are associated with.
|
|
||||||
- **AWS Session Token (optional)**: The AWS session token is an optional parameter. If you want, enter the details of your temporary session token.
|
|
||||||
- **Endpoint URL (optional)**: Your Glue connector will automatically determine the AWS QuickSight endpoint URL based on the region. You may override this behavior by entering a value to the endpoint URL.
|
|
||||||
|
|
||||||
{% /codeInfo %}
|
|
||||||
|
|
||||||
{% codeInfo srNumber=2 %}
|
{% codeInfo srNumber=2 %}
|
||||||
|
|
||||||
@ -150,13 +141,9 @@ source:
|
|||||||
config:
|
config:
|
||||||
type: QuickSight
|
type: QuickSight
|
||||||
```
|
```
|
||||||
```yaml {% srNumber=1 %}
|
|
||||||
awsConfig:
|
{% partial file="/v1.7/connectors/yaml/common/aws-config.md" /%}
|
||||||
awsAccessKeyId: KEY
|
|
||||||
awsSecretAccessKey: SECRET
|
|
||||||
awsRegion: us-east-2
|
|
||||||
awsSessionToken: Token
|
|
||||||
```
|
|
||||||
```yaml {% srNumber=2 %}
|
```yaml {% srNumber=2 %}
|
||||||
awsAccountId: <aws-account-id>
|
awsAccountId: <aws-account-id>
|
||||||
```
|
```
|
||||||
|
@ -172,100 +172,7 @@ This is a sample config for Athena:
|
|||||||
|
|
||||||
#### Source Configuration - Service Connection
|
#### Source Configuration - Service Connection
|
||||||
|
|
||||||
{% codeInfo srNumber=1 %}
|
{% partial file="/v1.7/connectors/yaml/common/aws-config-def.md" /%}
|
||||||
|
|
||||||
- **awsAccessKeyId** & **awsSecretAccessKey**: When you interact with AWS, you specify your AWS security credentials to verify who you are and whether you have
|
|
||||||
permission to access the resources that you are requesting. AWS uses the security credentials to authenticate and
|
|
||||||
authorize your requests ([docs](https://docs.aws.amazon.com/IAM/latest/UserGuide/security-creds.html)).
|
|
||||||
|
|
||||||
Access keys consist of two parts: An **access key ID** (for example, `AKIAIOSFODNN7EXAMPLE`), and a **secret access key** (for example, `wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY`).
|
|
||||||
|
|
||||||
You must use both the access key ID and secret access key together to authenticate your requests.
|
|
||||||
|
|
||||||
You can find further information on how to manage your access keys [here](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html).
|
|
||||||
|
|
||||||
{% /codeInfo %}
|
|
||||||
|
|
||||||
{% codeInfo srNumber=2 %}
|
|
||||||
**awsSessionToken**: If you are using temporary credentials to access your services, you will need to inform the AWS Access Key ID
|
|
||||||
and AWS Secrets Access Key. Also, these will include an AWS Session Token.
|
|
||||||
|
|
||||||
{% /codeInfo %}
|
|
||||||
|
|
||||||
{% codeInfo srNumber=3 %}
|
|
||||||
|
|
||||||
**awsRegion**: Each AWS Region is a separate geographic area in which AWS clusters data centers ([docs](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.RegionsAndAvailabilityZones.html)).
|
|
||||||
|
|
||||||
As AWS can have instances in multiple regions, we need to know the region the service you want reach belongs to.
|
|
||||||
|
|
||||||
Note that the AWS Region is the only required parameter when configuring a connection. When connecting to the
|
|
||||||
services programmatically, there are different ways in which we can extract and use the rest of AWS configurations.
|
|
||||||
|
|
||||||
You can find further information about configuring your credentials [here](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html#configuring-credentials).
|
|
||||||
|
|
||||||
{% /codeInfo %}
|
|
||||||
|
|
||||||
{% codeInfo srNumber=4 %}
|
|
||||||
|
|
||||||
**endPointURL**: To connect programmatically to an AWS service, you use an endpoint. An *endpoint* is the URL of the
|
|
||||||
entry point for an AWS web service. The AWS SDKs and the AWS Command Line Interface (AWS CLI) automatically use the
|
|
||||||
default endpoint for each service in an AWS Region. But you can specify an alternate endpoint for your API requests.
|
|
||||||
|
|
||||||
Find more information on [AWS service endpoints](https://docs.aws.amazon.com/general/latest/gr/rande.html).
|
|
||||||
|
|
||||||
{% /codeInfo %}
|
|
||||||
|
|
||||||
{% codeInfo srNumber=5 %}
|
|
||||||
|
|
||||||
**profileName**: A named profile is a collection of settings and credentials that you can apply to a AWS CLI command.
|
|
||||||
When you specify a profile to run a command, the settings and credentials are used to run that command.
|
|
||||||
Multiple named profiles can be stored in the config and credentials files.
|
|
||||||
|
|
||||||
You can inform this field if you'd like to use a profile other than `default`.
|
|
||||||
|
|
||||||
Find here more information about [Named profiles for the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-profiles.html).
|
|
||||||
|
|
||||||
{% /codeInfo %}
|
|
||||||
|
|
||||||
{% codeInfo srNumber=6 %}
|
|
||||||
|
|
||||||
**assumeRoleArn**: Typically, you use `AssumeRole` within your account or for cross-account access. In this field you'll set the
|
|
||||||
`ARN` (Amazon Resource Name) of the policy of the other account.
|
|
||||||
|
|
||||||
A user who wants to access a role in a different account must also have permissions that are delegated from the account
|
|
||||||
administrator. The administrator must attach a policy that allows the user to call `AssumeRole` for the `ARN` of the role in the other account.
|
|
||||||
|
|
||||||
This is a required field if you'd like to `AssumeRole`.
|
|
||||||
|
|
||||||
Find more information on [AssumeRole](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html).
|
|
||||||
|
|
||||||
{%note%}
|
|
||||||
When using Assume Role authentication, ensure you provide the following details:
|
|
||||||
- **AWS Region**: Specify the AWS region for your deployment.
|
|
||||||
- **Assume Role ARN**: Provide the ARN of the role in your AWS account that OpenMetadata will assume.
|
|
||||||
{%/note%}
|
|
||||||
|
|
||||||
{% /codeInfo %}
|
|
||||||
|
|
||||||
{% codeInfo srNumber=7 %}
|
|
||||||
|
|
||||||
**assumeRoleSessionName**: An identifier for the assumed role session. Use the role session name to uniquely identify a session when the same role
|
|
||||||
is assumed by different principals or for different reasons.
|
|
||||||
|
|
||||||
By default, we'll use the name `OpenMetadataSession`.
|
|
||||||
|
|
||||||
Find more information about the [Role Session Name](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html#:~:text=An%20identifier%20for%20the%20assumed%20role%20session.).
|
|
||||||
|
|
||||||
{% /codeInfo %}
|
|
||||||
|
|
||||||
{% codeInfo srNumber=8 %}
|
|
||||||
|
|
||||||
**assumeRoleSourceIdentity**: The source identity specified by the principal that is calling the `AssumeRole` operation. You can use source identity
|
|
||||||
information in AWS CloudTrail logs to determine who took actions with a role.
|
|
||||||
|
|
||||||
Find more information about [Source Identity](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html#:~:text=Required%3A%20No-,SourceIdentity,-The%20source%20identity).
|
|
||||||
|
|
||||||
{% /codeInfo %}
|
|
||||||
|
|
||||||
{% codeInfo srNumber=9 %}
|
{% codeInfo srNumber=9 %}
|
||||||
|
|
||||||
@ -312,31 +219,9 @@ source:
|
|||||||
type: Athena
|
type: Athena
|
||||||
awsConfig:
|
awsConfig:
|
||||||
```
|
```
|
||||||
```yaml {% srNumber=1 %}
|
|
||||||
awsAccessKeyId: KEY
|
{% partial file="/v1.7/connectors/yaml/common/aws-config.md" /%}
|
||||||
awsSecretAccessKey: SECRET
|
|
||||||
```
|
|
||||||
```yaml {% srNumber=2 %}
|
|
||||||
# awsSessionToken: TOKEN
|
|
||||||
```
|
|
||||||
```yaml {% srNumber=3 %}
|
|
||||||
awsRegion: us-east-2
|
|
||||||
```
|
|
||||||
```yaml {% srNumber=4 %}
|
|
||||||
# endPointURL: https://athena.us-east-2.amazonaws.com/custom
|
|
||||||
```
|
|
||||||
```yaml {% srNumber=5 %}
|
|
||||||
# profileName: profile
|
|
||||||
```
|
|
||||||
```yaml {% srNumber=6 %}
|
|
||||||
# assumeRoleArn: "arn:partition:service:region:account:resource"
|
|
||||||
```
|
|
||||||
```yaml {% srNumber=7 %}
|
|
||||||
# assumeRoleSessionName: session
|
|
||||||
```
|
|
||||||
```yaml {% srNumber=8 %}
|
|
||||||
# assumeRoleSourceIdentity: identity
|
|
||||||
```
|
|
||||||
```yaml {% srNumber=9 %}
|
```yaml {% srNumber=9 %}
|
||||||
s3StagingDir: s3 directory for datasource
|
s3StagingDir: s3 directory for datasource
|
||||||
```
|
```
|
||||||
|
@ -78,35 +78,7 @@ This is a sample config for DynamoDB:
|
|||||||
|
|
||||||
#### Source Configuration - Service Connection
|
#### Source Configuration - Service Connection
|
||||||
|
|
||||||
{% codeInfo srNumber=1 %}
|
{% partial file="/v1.7/connectors/yaml/common/aws-config-def.md" /%}
|
||||||
|
|
||||||
**awsAccessKeyId**: Enter your secure access key ID for your DynamoDB connection. The specified key ID should be authorized to read all databases you want to include in the metadata ingestion workflow.
|
|
||||||
|
|
||||||
{% /codeInfo %}
|
|
||||||
|
|
||||||
{% codeInfo srNumber=2 %}
|
|
||||||
|
|
||||||
**awsSecretAccessKey**: Enter the Secret Access Key (the passcode key pair to the key ID from above).
|
|
||||||
|
|
||||||
{% /codeInfo %}
|
|
||||||
|
|
||||||
{% codeInfo srNumber=3 %}
|
|
||||||
|
|
||||||
**awsSessionToken**: The AWS session token is an optional parameter. If you want, enter the details of your temporary session token.
|
|
||||||
|
|
||||||
{% /codeInfo %}
|
|
||||||
|
|
||||||
{% codeInfo srNumber=4 %}
|
|
||||||
|
|
||||||
**awsRegion**: Enter the location of the amazon cluster that your data and account are associated with.
|
|
||||||
|
|
||||||
{% /codeInfo %}
|
|
||||||
|
|
||||||
{% codeInfo srNumber=5 %}
|
|
||||||
|
|
||||||
**endPointURL**: Your DynamoDB connector will automatically determine the AWS DynamoDB endpoint URL based on the region. You may override this behavior by entering a value to the endpoint URL.
|
|
||||||
|
|
||||||
{% /codeInfo %}
|
|
||||||
|
|
||||||
{% codeInfo srNumber=6 %}
|
{% codeInfo srNumber=6 %}
|
||||||
|
|
||||||
@ -149,21 +121,9 @@ source:
|
|||||||
type: DynamoDB
|
type: DynamoDB
|
||||||
awsConfig:
|
awsConfig:
|
||||||
```
|
```
|
||||||
```yaml {% srNumber=1 %}
|
|
||||||
awsAccessKeyId: aws_access_key_id
|
{% partial file="/v1.7/connectors/yaml/common/aws-config.md" /%}
|
||||||
```
|
|
||||||
```yaml {% srNumber=2 %}
|
|
||||||
awsSecretAccessKey: aws_secret_access_key
|
|
||||||
```
|
|
||||||
```yaml {% srNumber=3 %}
|
|
||||||
awsSessionToken: AWS Session Token
|
|
||||||
```
|
|
||||||
```yaml {% srNumber=4 %}
|
|
||||||
awsRegion: aws region
|
|
||||||
```
|
|
||||||
```yaml {% srNumber=5 %}
|
|
||||||
endPointURL: https://dynamodb.<region_name>.amazonaws.com
|
|
||||||
```
|
|
||||||
```yaml {% srNumber=6 %}
|
```yaml {% srNumber=6 %}
|
||||||
database: custom_database_name
|
database: custom_database_name
|
||||||
```
|
```
|
||||||
|
@ -58,35 +58,7 @@ This is a sample config for Glue:
|
|||||||
|
|
||||||
#### Source Configuration - Service Connection
|
#### Source Configuration - Service Connection
|
||||||
|
|
||||||
{% codeInfo srNumber=1 %}
|
{% partial file="/v1.7/connectors/yaml/common/aws-config-def.md" /%}
|
||||||
|
|
||||||
**awsAccessKeyId**: Enter your secure access key ID for your Glue connection. The specified key ID should be authorized to read all databases you want to include in the metadata ingestion workflow.
|
|
||||||
|
|
||||||
{% /codeInfo %}
|
|
||||||
|
|
||||||
{% codeInfo srNumber=2 %}
|
|
||||||
|
|
||||||
**awsSecretAccessKey**: Enter the Secret Access Key (the passcode key pair to the key ID from above).
|
|
||||||
|
|
||||||
{% /codeInfo %}
|
|
||||||
|
|
||||||
{% codeInfo srNumber=3 %}
|
|
||||||
|
|
||||||
**awsRegion**: Enter the location of the amazon cluster that your data and account are associated with.
|
|
||||||
|
|
||||||
{% /codeInfo %}
|
|
||||||
|
|
||||||
{% codeInfo srNumber=4 %}
|
|
||||||
|
|
||||||
**endPointURL**: Your Glue connector will automatically determine the AWS Glue endpoint URL based on the region. You may override this behavior by entering a value to the endpoint URL.
|
|
||||||
|
|
||||||
{% /codeInfo %}
|
|
||||||
|
|
||||||
{% codeInfo srNumber=5 %}
|
|
||||||
|
|
||||||
**awsSessionToken**: The AWS session token is an optional parameter. If you want, enter the details of your temporary session token.
|
|
||||||
|
|
||||||
{% /codeInfo %}
|
|
||||||
|
|
||||||
{% codeInfo srNumber=6 %}
|
{% codeInfo srNumber=6 %}
|
||||||
|
|
||||||
@ -130,21 +102,9 @@ source:
|
|||||||
type: Glue
|
type: Glue
|
||||||
awsConfig:
|
awsConfig:
|
||||||
```
|
```
|
||||||
```yaml {% srNumber=1 %}
|
|
||||||
awsAccessKeyId: KEY
|
{% partial file="/v1.7/connectors/yaml/common/aws-config.md" /%}
|
||||||
```
|
|
||||||
```yaml {% srNumber=2 %}
|
|
||||||
awsSecretAccessKey: SECRET
|
|
||||||
```
|
|
||||||
```yaml {% srNumber=3 %}
|
|
||||||
awsRegion: us-east-2
|
|
||||||
```
|
|
||||||
```yaml {% srNumber=4 %}
|
|
||||||
# endPointURL: https://glue.us-east-2.amazonaws.com/
|
|
||||||
```
|
|
||||||
```yaml {% srNumber=5 %}
|
|
||||||
# awsSessionToken: TOKEN
|
|
||||||
```
|
|
||||||
```yaml {% srNumber=6 %}
|
```yaml {% srNumber=6 %}
|
||||||
databaseName: database_name
|
databaseName: database_name
|
||||||
```
|
```
|
||||||
|
@ -90,13 +90,11 @@ The workflow is modeled around the following JSON Schema.
|
|||||||
|
|
||||||
{% codeInfo srNumber=1 %}
|
{% codeInfo srNumber=1 %}
|
||||||
|
|
||||||
* **awsAccessKeyId**: Enter your secure access key ID for your DynamoDB connection. The specified key ID should be authorized to read all databases you want to include in the metadata ingestion workflow.
|
|
||||||
* **awsSecretAccessKey**: Enter the Secret Access Key (the passcode key pair to the key ID from above).
|
|
||||||
* **awsRegion**: Specify the region in which your DynamoDB is located. This setting is required even if you have configured a local AWS profile.
|
|
||||||
* **schemaFilterPattern** and **tableFilterPattern**: Note that the `schemaFilterPattern` and `tableFilterPattern` both support regex as `include` or `exclude`. E.g.,
|
* **schemaFilterPattern** and **tableFilterPattern**: Note that the `schemaFilterPattern` and `tableFilterPattern` both support regex as `include` or `exclude`. E.g.,
|
||||||
|
|
||||||
{% /codeInfo %}
|
{% /codeInfo %}
|
||||||
|
|
||||||
|
{% partial file="/v1.7/connectors/yaml/common/aws-config-def.md" /%}
|
||||||
|
|
||||||
{% partial file="/v1.7/connectors/yaml/database/source-config-def.md" /%}
|
{% partial file="/v1.7/connectors/yaml/database/source-config-def.md" /%}
|
||||||
|
|
||||||
@ -120,9 +118,11 @@ source:
|
|||||||
```yaml {% srNumber=1 %}
|
```yaml {% srNumber=1 %}
|
||||||
configSource:
|
configSource:
|
||||||
securityConfig:
|
securityConfig:
|
||||||
awsAccessKeyId: aws access key id
|
```
|
||||||
awsSecretAccessKey: aws secret access key
|
|
||||||
awsRegion: aws region
|
{% partial file="/v1.7/connectors/yaml/common/aws-config.md" /%}
|
||||||
|
|
||||||
|
```yaml {% srNumber=1 %}
|
||||||
bucketName: bucket name
|
bucketName: bucket name
|
||||||
prefix: prefix
|
prefix: prefix
|
||||||
```
|
```
|
||||||
|
@ -46,38 +46,7 @@ In this configuration we will be fetching the dbt `manifest.json`, `catalog.json
|
|||||||
- **dbtConfigType**: s3
|
- **dbtConfigType**: s3
|
||||||
{% /codeInfo %}
|
{% /codeInfo %}
|
||||||
|
|
||||||
{% codeInfo srNumber=1 %}
|
{% partial file="/v1.7/connectors/yaml/common/aws-config-def.md" /%}
|
||||||
|
|
||||||
- **awsAccessKeyId** & **awsSecretAccessKey**: When you interact with AWS, you specify your AWS security credentials to verify who you are and whether you have
|
|
||||||
permission to access the resources that you are requesting. AWS uses the security credentials to authenticate and
|
|
||||||
authorize your requests ([docs](https://docs.aws.amazon.com/IAM/latest/UserGuide/security-creds.html)).
|
|
||||||
|
|
||||||
Access keys consist of two parts: An **access key ID** (for example, `AKIAIOSFODNN7EXAMPLE`), and a **secret access key** (for example, `wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY`).
|
|
||||||
|
|
||||||
You must use both the access key ID and secret access key together to authenticate your requests.
|
|
||||||
|
|
||||||
You can find further information on how to manage your access keys [here](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html).
|
|
||||||
|
|
||||||
{% /codeInfo %}
|
|
||||||
|
|
||||||
{% codeInfo srNumber=2 %}
|
|
||||||
**awsSessionToken**: If you are using temporary credentials to access your services, you will need to inform the AWS Access Key ID
|
|
||||||
and AWS Secrets Access Key. Also, these will include an AWS Session Token.
|
|
||||||
|
|
||||||
{% /codeInfo %}
|
|
||||||
|
|
||||||
{% codeInfo srNumber=3 %}
|
|
||||||
|
|
||||||
**awsRegion**: Each AWS Region is a separate geographic area in which AWS clusters data centers ([docs](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.RegionsAndAvailabilityZones.html)).
|
|
||||||
|
|
||||||
As AWS can have instances in multiple regions, we need to know the region the service you want reach belongs to.
|
|
||||||
|
|
||||||
Note that the AWS Region is the only required parameter when configuring a connection. When connecting to the
|
|
||||||
services programmatically, there are different ways in which we can extract and use the rest of AWS configurations.
|
|
||||||
|
|
||||||
You can find further information about configuring your credentials [here](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html#configuring-credentials).
|
|
||||||
|
|
||||||
{% /codeInfo %}
|
|
||||||
|
|
||||||
{% partial file="/v1.7/connectors/yaml/dbt/dbt-prefix-def.md" /%}
|
{% partial file="/v1.7/connectors/yaml/dbt/dbt-prefix-def.md" /%}
|
||||||
|
|
||||||
@ -105,16 +74,10 @@ source:
|
|||||||
```
|
```
|
||||||
```yaml {% srNumber=1 %}
|
```yaml {% srNumber=1 %}
|
||||||
dbtSecurityConfig:
|
dbtSecurityConfig:
|
||||||
awsAccessKeyId: AWS_ACCESS_KEY_ID
|
|
||||||
awsSecretAccessKey: AWS_SECRET_ACCESS_KEY
|
|
||||||
```
|
|
||||||
```yaml {% srNumber=2 %}
|
|
||||||
awsSessionToken: AWS_SESSION_TOKEN
|
|
||||||
```
|
|
||||||
```yaml {% srNumber=3 %}
|
|
||||||
awsRegion: AWS_REGION
|
|
||||||
```
|
```
|
||||||
|
|
||||||
|
{% partial file="/v1.7/connectors/yaml/common/aws-config.md" /%}
|
||||||
|
|
||||||
{% partial file="/v1.7/connectors/yaml/dbt/dbt-prefix.md" /%}
|
{% partial file="/v1.7/connectors/yaml/dbt/dbt-prefix.md" /%}
|
||||||
|
|
||||||
{% partial file="/v1.7/connectors/yaml/dbt/source-config.md" /%}
|
{% partial file="/v1.7/connectors/yaml/dbt/source-config.md" /%}
|
||||||
|
@ -80,100 +80,7 @@ This is a sample config for Kinesis:
|
|||||||
|
|
||||||
#### Source Configuration - Service Connection
|
#### Source Configuration - Service Connection
|
||||||
|
|
||||||
{% codeInfo srNumber=1 %}
|
{% partial file="/v1.7/connectors/yaml/common/aws-config-def.md" /%}
|
||||||
|
|
||||||
- **awsAccessKeyId** & **awsSecretAccessKey**: When you interact with AWS, you specify your AWS security credentials to verify who you are and whether you have
|
|
||||||
permission to access the resources that you are requesting. AWS uses the security credentials to authenticate and
|
|
||||||
authorize your requests ([docs](https://docs.aws.amazon.com/IAM/latest/UserGuide/security-creds.html)).
|
|
||||||
|
|
||||||
Access keys consist of two parts: An **access key ID** (for example, `AKIAIOSFODNN7EXAMPLE`), and a **secret access key** (for example, `wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY`).
|
|
||||||
|
|
||||||
You must use both the access key ID and secret access key together to authenticate your requests.
|
|
||||||
|
|
||||||
You can find further information on how to manage your access keys [here](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html).
|
|
||||||
|
|
||||||
{% /codeInfo %}
|
|
||||||
|
|
||||||
{% codeInfo srNumber=2 %}
|
|
||||||
**awsSessionToken**: If you are using temporary credentials to access your services, you will need to inform the AWS Access Key ID
|
|
||||||
and AWS Secrets Access Key. Also, these will include an AWS Session Token.
|
|
||||||
|
|
||||||
{% /codeInfo %}
|
|
||||||
|
|
||||||
{% codeInfo srNumber=3 %}
|
|
||||||
|
|
||||||
**awsRegion**: Each AWS Region is a separate geographic area in which AWS clusters data centers ([docs](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.RegionsAndAvailabilityZones.html)).
|
|
||||||
|
|
||||||
As AWS can have instances in multiple regions, we need to know the region the service you want reach belongs to.
|
|
||||||
|
|
||||||
Note that the AWS Region is the only required parameter when configuring a connection. When connecting to the
|
|
||||||
services programmatically, there are different ways in which we can extract and use the rest of AWS configurations.
|
|
||||||
|
|
||||||
You can find further information about configuring your credentials [here](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html#configuring-credentials).
|
|
||||||
|
|
||||||
{% /codeInfo %}
|
|
||||||
|
|
||||||
{% codeInfo srNumber=4 %}
|
|
||||||
|
|
||||||
**endPointURL**: To connect programmatically to an AWS service, you use an endpoint. An *endpoint* is the URL of the
|
|
||||||
entry point for an AWS web service. The AWS SDKs and the AWS Command Line Interface (AWS CLI) automatically use the
|
|
||||||
default endpoint for each service in an AWS Region. But you can specify an alternate endpoint for your API requests.
|
|
||||||
|
|
||||||
Find more information on [AWS service endpoints](https://docs.aws.amazon.com/general/latest/gr/rande.html).
|
|
||||||
|
|
||||||
{% /codeInfo %}
|
|
||||||
|
|
||||||
{% codeInfo srNumber=5 %}
|
|
||||||
|
|
||||||
**profileName**: A named profile is a collection of settings and credentials that you can apply to a AWS CLI command.
|
|
||||||
When you specify a profile to run a command, the settings and credentials are used to run that command.
|
|
||||||
Multiple named profiles can be stored in the config and credentials files.
|
|
||||||
|
|
||||||
You can inform this field if you'd like to use a profile other than `default`.
|
|
||||||
|
|
||||||
Find here more information about [Named profiles for the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-profiles.html).
|
|
||||||
|
|
||||||
{% /codeInfo %}
|
|
||||||
|
|
||||||
{% codeInfo srNumber=6 %}
|
|
||||||
|
|
||||||
**assumeRoleArn**: Typically, you use `AssumeRole` within your account or for cross-account access. In this field you'll set the
|
|
||||||
`ARN` (Amazon Resource Name) of the policy of the other account.
|
|
||||||
|
|
||||||
A user who wants to access a role in a different account must also have permissions that are delegated from the account
|
|
||||||
administrator. The administrator must attach a policy that allows the user to call `AssumeRole` for the `ARN` of the role in the other account.
|
|
||||||
|
|
||||||
This is a required field if you'd like to `AssumeRole`.
|
|
||||||
|
|
||||||
Find more information on [AssumeRole](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html).
|
|
||||||
|
|
||||||
{%note%}
|
|
||||||
When using Assume Role authentication, ensure you provide the following details:
|
|
||||||
- **AWS Region**: Specify the AWS region for your deployment.
|
|
||||||
- **Assume Role ARN**: Provide the ARN of the role in your AWS account that OpenMetadata will assume.
|
|
||||||
{%/note%}
|
|
||||||
|
|
||||||
{% /codeInfo %}
|
|
||||||
|
|
||||||
{% codeInfo srNumber=7 %}
|
|
||||||
|
|
||||||
**assumeRoleSessionName**: An identifier for the assumed role session. Use the role session name to uniquely identify a session when the same role
|
|
||||||
is assumed by different principals or for different reasons.
|
|
||||||
|
|
||||||
By default, we'll use the name `OpenMetadataSession`.
|
|
||||||
|
|
||||||
Find more information about the [Role Session Name](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html#:~:text=An%20identifier%20for%20the%20assumed%20role%20session.).
|
|
||||||
|
|
||||||
{% /codeInfo %}
|
|
||||||
|
|
||||||
{% codeInfo srNumber=8 %}
|
|
||||||
|
|
||||||
**assumeRoleSourceIdentity**: The source identity specified by the principal that is calling the `AssumeRole` operation. You can use source identity
|
|
||||||
information in AWS CloudTrail logs to determine who took actions with a role.
|
|
||||||
|
|
||||||
Find more information about [Source Identity](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html#:~:text=Required%3A%20No-,SourceIdentity,-The%20source%20identity).
|
|
||||||
|
|
||||||
{% /codeInfo %}
|
|
||||||
|
|
||||||
{% partial file="/v1.7/connectors/yaml/messaging/source-config-def.md" /%}
|
{% partial file="/v1.7/connectors/yaml/messaging/source-config-def.md" /%}
|
||||||
|
|
||||||
@ -195,31 +102,8 @@ source:
|
|||||||
type: Kinesis
|
type: Kinesis
|
||||||
awsConfig:
|
awsConfig:
|
||||||
```
|
```
|
||||||
```yaml {% srNumber=1 %}
|
|
||||||
awsAccessKeyId: KEY
|
{% partial file="/v1.7/connectors/yaml/common/aws-config.md" /%}
|
||||||
awsSecretAccessKey: SECRET
|
|
||||||
```
|
|
||||||
```yaml {% srNumber=2 %}
|
|
||||||
# awsSessionToken: TOKEN
|
|
||||||
```
|
|
||||||
```yaml {% srNumber=3 %}
|
|
||||||
awsRegion: us-east-2
|
|
||||||
```
|
|
||||||
```yaml {% srNumber=4 %}
|
|
||||||
# endPointURL: https://athena.us-east-2.amazonaws.com/custom
|
|
||||||
```
|
|
||||||
```yaml {% srNumber=5 %}
|
|
||||||
# profileName: profile
|
|
||||||
```
|
|
||||||
```yaml {% srNumber=6 %}
|
|
||||||
# assumeRoleArn: "arn:partition:service:region:account:resource"
|
|
||||||
```
|
|
||||||
```yaml {% srNumber=7 %}
|
|
||||||
# assumeRoleSessionName: session
|
|
||||||
```
|
|
||||||
```yaml {% srNumber=8 %}
|
|
||||||
# assumeRoleSourceIdentity: identity
|
|
||||||
```
|
|
||||||
|
|
||||||
{% partial file="/v1.7/connectors/yaml/messaging/source-config.md" /%}
|
{% partial file="/v1.7/connectors/yaml/messaging/source-config.md" /%}
|
||||||
|
|
||||||
|
@ -78,100 +78,7 @@ This is a sample config for Sagemaker:
|
|||||||
|
|
||||||
#### Source Configuration - Service Connection
|
#### Source Configuration - Service Connection
|
||||||
|
|
||||||
{% codeInfo srNumber=1 %}
|
{% partial file="/v1.7/connectors/yaml/common/aws-config-def.md" /%}
|
||||||
|
|
||||||
- **awsAccessKeyId** & **awsSecretAccessKey**: When you interact with AWS, you specify your AWS security credentials to verify who you are and whether you have
|
|
||||||
permission to access the resources that you are requesting. AWS uses the security credentials to authenticate and
|
|
||||||
authorize your requests ([docs](https://docs.aws.amazon.com/IAM/latest/UserGuide/security-creds.html)).
|
|
||||||
|
|
||||||
Access keys consist of two parts: An **access key ID** (for example, `AKIAIOSFODNN7EXAMPLE`), and a **secret access key** (for example, `wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY`).
|
|
||||||
|
|
||||||
You must use both the access key ID and secret access key together to authenticate your requests.
|
|
||||||
|
|
||||||
You can find further information on how to manage your access keys [here](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html).
|
|
||||||
|
|
||||||
{% /codeInfo %}
|
|
||||||
|
|
||||||
{% codeInfo srNumber=2 %}
|
|
||||||
**awsSessionToken**: If you are using temporary credentials to access your services, you will need to inform the AWS Access Key ID
|
|
||||||
and AWS Secrets Access Key. Also, these will include an AWS Session Token.
|
|
||||||
|
|
||||||
{% /codeInfo %}
|
|
||||||
|
|
||||||
{% codeInfo srNumber=3 %}
|
|
||||||
|
|
||||||
**awsRegion**: Each AWS Region is a separate geographic area in which AWS clusters data centers ([docs](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.RegionsAndAvailabilityZones.html)).
|
|
||||||
|
|
||||||
As AWS can have instances in multiple regions, we need to know the region the service you want reach belongs to.
|
|
||||||
|
|
||||||
Note that the AWS Region is the only required parameter when configuring a connection. When connecting to the
|
|
||||||
services programmatically, there are different ways in which we can extract and use the rest of AWS configurations.
|
|
||||||
|
|
||||||
You can find further information about configuring your credentials [here](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html#configuring-credentials).
|
|
||||||
|
|
||||||
{% /codeInfo %}
|
|
||||||
|
|
||||||
{% codeInfo srNumber=4 %}
|
|
||||||
|
|
||||||
**endPointURL**: To connect programmatically to an AWS service, you use an endpoint. An *endpoint* is the URL of the
|
|
||||||
entry point for an AWS web service. The AWS SDKs and the AWS Command Line Interface (AWS CLI) automatically use the
|
|
||||||
default endpoint for each service in an AWS Region. But you can specify an alternate endpoint for your API requests.
|
|
||||||
|
|
||||||
Find more information on [AWS service endpoints](https://docs.aws.amazon.com/general/latest/gr/rande.html).
|
|
||||||
|
|
||||||
{% /codeInfo %}
|
|
||||||
|
|
||||||
{% codeInfo srNumber=5 %}
|
|
||||||
|
|
||||||
**profileName**: A named profile is a collection of settings and credentials that you can apply to a AWS CLI command.
|
|
||||||
When you specify a profile to run a command, the settings and credentials are used to run that command.
|
|
||||||
Multiple named profiles can be stored in the config and credentials files.
|
|
||||||
|
|
||||||
You can inform this field if you'd like to use a profile other than `default`.
|
|
||||||
|
|
||||||
Find here more information about [Named profiles for the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-profiles.html).
|
|
||||||
|
|
||||||
{% /codeInfo %}
|
|
||||||
|
|
||||||
{% codeInfo srNumber=6 %}
|
|
||||||
|
|
||||||
**assumeRoleArn**: Typically, you use `AssumeRole` within your account or for cross-account access. In this field you'll set the
|
|
||||||
`ARN` (Amazon Resource Name) of the policy of the other account.
|
|
||||||
|
|
||||||
A user who wants to access a role in a different account must also have permissions that are delegated from the account
|
|
||||||
administrator. The administrator must attach a policy that allows the user to call `AssumeRole` for the `ARN` of the role in the other account.
|
|
||||||
|
|
||||||
This is a required field if you'd like to `AssumeRole`.
|
|
||||||
|
|
||||||
Find more information on [AssumeRole](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html).
|
|
||||||
|
|
||||||
{%note%}
|
|
||||||
When using Assume Role authentication, ensure you provide the following details:
|
|
||||||
- **AWS Region**: Specify the AWS region for your deployment.
|
|
||||||
- **Assume Role ARN**: Provide the ARN of the role in your AWS account that OpenMetadata will assume.
|
|
||||||
{%/note%}
|
|
||||||
|
|
||||||
{% /codeInfo %}
|
|
||||||
|
|
||||||
{% codeInfo srNumber=7 %}
|
|
||||||
|
|
||||||
**assumeRoleSessionName**: An identifier for the assumed role session. Use the role session name to uniquely identify a session when the same role
|
|
||||||
is assumed by different principals or for different reasons.
|
|
||||||
|
|
||||||
By default, we'll use the name `OpenMetadataSession`.
|
|
||||||
|
|
||||||
Find more information about the [Role Session Name](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html#:~:text=An%20identifier%20for%20the%20assumed%20role%20session.).
|
|
||||||
|
|
||||||
{% /codeInfo %}
|
|
||||||
|
|
||||||
{% codeInfo srNumber=8 %}
|
|
||||||
|
|
||||||
**assumeRoleSourceIdentity**: The source identity specified by the principal that is calling the `AssumeRole` operation. You can use source identity
|
|
||||||
information in AWS CloudTrail logs to determine who took actions with a role.
|
|
||||||
|
|
||||||
Find more information about [Source Identity](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html#:~:text=Required%3A%20No-,SourceIdentity,-The%20source%20identity).
|
|
||||||
|
|
||||||
{% /codeInfo %}
|
|
||||||
|
|
||||||
{% partial file="/v1.7/connectors/yaml/ml-model/source-config-def.md" /%}
|
{% partial file="/v1.7/connectors/yaml/ml-model/source-config-def.md" /%}
|
||||||
|
|
||||||
@ -193,31 +100,8 @@ source:
|
|||||||
type: SageMaker
|
type: SageMaker
|
||||||
awsConfig:
|
awsConfig:
|
||||||
```
|
```
|
||||||
```yaml {% srNumber=1 %}
|
|
||||||
awsAccessKeyId: KEY
|
{% partial file="/v1.7/connectors/yaml/common/aws-config.md" /%}
|
||||||
awsSecretAccessKey: SECRET
|
|
||||||
```
|
|
||||||
```yaml {% srNumber=2 %}
|
|
||||||
# awsSessionToken: TOKEN
|
|
||||||
```
|
|
||||||
```yaml {% srNumber=3 %}
|
|
||||||
awsRegion: us-east-2
|
|
||||||
```
|
|
||||||
```yaml {% srNumber=4 %}
|
|
||||||
# endPointURL: https://athena.us-east-2.amazonaws.com/custom
|
|
||||||
```
|
|
||||||
```yaml {% srNumber=5 %}
|
|
||||||
# profileName: profile
|
|
||||||
```
|
|
||||||
```yaml {% srNumber=6 %}
|
|
||||||
# assumeRoleArn: "arn:partition:service:region:account:resource"
|
|
||||||
```
|
|
||||||
```yaml {% srNumber=7 %}
|
|
||||||
# assumeRoleSessionName: session
|
|
||||||
```
|
|
||||||
```yaml {% srNumber=8 %}
|
|
||||||
# assumeRoleSourceIdentity: identity
|
|
||||||
```
|
|
||||||
|
|
||||||
{% partial file="/v1.7/connectors/yaml/ml-model/source-config.md" /%}
|
{% partial file="/v1.7/connectors/yaml/ml-model/source-config.md" /%}
|
||||||
|
|
||||||
|
@ -63,58 +63,7 @@ This is a sample config for Glue:
|
|||||||
|
|
||||||
#### Source Configuration - Service Connection
|
#### Source Configuration - Service Connection
|
||||||
|
|
||||||
{% codeInfo srNumber=1 %}
|
{% partial file="/v1.7/connectors/yaml/common/aws-config-def.md" /%}
|
||||||
|
|
||||||
|
|
||||||
**awsAccessKeyId**: Enter your secure access key ID for your Glue connection. The specified key ID should be
|
|
||||||
authorized to read all databases you want to include in the metadata ingestion workflow.
|
|
||||||
|
|
||||||
{% /codeInfo %}
|
|
||||||
|
|
||||||
{% codeInfo srNumber=2 %}
|
|
||||||
|
|
||||||
**awsSecretAccessKey**: Enter the Secret Access Key (the passcode key pair to the key ID from above).
|
|
||||||
|
|
||||||
|
|
||||||
{% /codeInfo %}
|
|
||||||
|
|
||||||
{% codeInfo srNumber=3 %}
|
|
||||||
|
|
||||||
**awsRegion**: Enter the location of the amazon cluster that your data and account are associated with.
|
|
||||||
|
|
||||||
{% /codeInfo %}
|
|
||||||
|
|
||||||
{% codeInfo srNumber=4 %}
|
|
||||||
|
|
||||||
**awsSessionToken**: The AWS session token is an optional parameter. If you want, enter the details of your temporary
|
|
||||||
session token.
|
|
||||||
|
|
||||||
{% /codeInfo %}
|
|
||||||
|
|
||||||
{% codeInfo srNumber=5 %}
|
|
||||||
|
|
||||||
**endPointURL**: Your Glue connector will automatically determine the AWS Glue endpoint URL based on the region. You
|
|
||||||
may override this behavior by entering a value to the endpoint URL.
|
|
||||||
|
|
||||||
|
|
||||||
{% /codeInfo %}
|
|
||||||
|
|
||||||
{% codeInfo srNumber=6 %}
|
|
||||||
**profileName**: The name of a profile to use with the boto session.
|
|
||||||
{% /codeInfo %}
|
|
||||||
|
|
||||||
{% codeInfo srNumber=7 %}
|
|
||||||
**assumeRoleArn**: The Amazon Resource Name (ARN) of the role to assume. Required Field in case of Assume Role.
|
|
||||||
{% /codeInfo %}
|
|
||||||
|
|
||||||
{% codeInfo srNumber=8 %}
|
|
||||||
**assumeRoleSessionName**: An identifier for the assumed role session. Use the role session name to uniquely identify a session when the same role is assumed by different principals or for different reasons. Required Field in case of Assume Role.
|
|
||||||
{% /codeInfo %}
|
|
||||||
|
|
||||||
{% codeInfo srNumber=9 %}
|
|
||||||
**assumeRoleSourceIdentity**: The Amazon Resource Name (ARN) of the role to assume. Optional Field in case of Assume Role.
|
|
||||||
{% /codeInfo %}
|
|
||||||
|
|
||||||
|
|
||||||
{% partial file="/v1.6/connectors/yaml/pipeline/source-config-def.md" /%}
|
{% partial file="/v1.6/connectors/yaml/pipeline/source-config-def.md" /%}
|
||||||
|
|
||||||
@ -135,34 +84,7 @@ source:
|
|||||||
type: GluePipeline
|
type: GluePipeline
|
||||||
awsConfig:
|
awsConfig:
|
||||||
```
|
```
|
||||||
```yaml {% srNumber=1 %}
|
{% partial file="/v1.7/connectors/yaml/common/aws-config.md" /%}
|
||||||
awsAccessKeyId: KEY
|
|
||||||
```
|
|
||||||
```yaml {% srNumber=2 %}
|
|
||||||
awsSecretAccessKey: SECRET
|
|
||||||
```
|
|
||||||
```yaml {% srNumber=3 %}
|
|
||||||
awsRegion: us-east-2
|
|
||||||
```
|
|
||||||
```yaml {% srNumber=4 %}
|
|
||||||
# awsSessionToken: TOKEN
|
|
||||||
```
|
|
||||||
```yaml {% srNumber=5 %}
|
|
||||||
# endPointURL: https://glue.us-east-2.amazonaws.com/
|
|
||||||
```
|
|
||||||
```yaml {% srNumber=6 %}
|
|
||||||
# profileName: ""
|
|
||||||
```
|
|
||||||
```yaml {% srNumber=7 %}
|
|
||||||
# assumeRoleArn: ""
|
|
||||||
```
|
|
||||||
```yaml {% srNumber=8 %}
|
|
||||||
# assumeRoleSessionName: OpenMetadataSession
|
|
||||||
```
|
|
||||||
```yaml {% srNumber=9 %}
|
|
||||||
# assumeRoleSourceIdentity : ""
|
|
||||||
```
|
|
||||||
|
|
||||||
|
|
||||||
{% partial file="/v1.6/connectors/yaml/pipeline/source-config.md" /%}
|
{% partial file="/v1.6/connectors/yaml/pipeline/source-config.md" /%}
|
||||||
|
|
||||||
|
@ -128,100 +128,7 @@ This is a sample config for Athena:
|
|||||||
|
|
||||||
#### Source Configuration - Service Connection
|
#### Source Configuration - Service Connection
|
||||||
|
|
||||||
{% codeInfo srNumber=1 %}
|
{% partial file="/v1.7/connectors/yaml/common/aws-config-def.md" /%}
|
||||||
|
|
||||||
- **awsAccessKeyId** & **awsSecretAccessKey**: When you interact with AWS, you specify your AWS security credentials to verify who you are and whether you have
|
|
||||||
permission to access the resources that you are requesting. AWS uses the security credentials to authenticate and
|
|
||||||
authorize your requests ([docs](https://docs.aws.amazon.com/IAM/latest/UserGuide/security-creds.html)).
|
|
||||||
|
|
||||||
Access keys consist of two parts: An **access key ID** (for example, `AKIAIOSFODNN7EXAMPLE`), and a **secret access key** (for example, `wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY`).
|
|
||||||
|
|
||||||
You must use both the access key ID and secret access key together to authenticate your requests.
|
|
||||||
|
|
||||||
You can find further information on how to manage your access keys [here](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html).
|
|
||||||
|
|
||||||
{% /codeInfo %}
|
|
||||||
|
|
||||||
{% codeInfo srNumber=2 %}
|
|
||||||
**awsSessionToken**: If you are using temporary credentials to access your services, you will need to inform the AWS Access Key ID
|
|
||||||
and AWS Secrets Access Key. Also, these will include an AWS Session Token.
|
|
||||||
|
|
||||||
{% /codeInfo %}
|
|
||||||
|
|
||||||
{% codeInfo srNumber=3 %}
|
|
||||||
|
|
||||||
**awsRegion**: Each AWS Region is a separate geographic area in which AWS clusters data centers ([docs](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.RegionsAndAvailabilityZones.html)).
|
|
||||||
|
|
||||||
As AWS can have instances in multiple regions, we need to know the region the service you want reach belongs to.
|
|
||||||
|
|
||||||
Note that the AWS Region is the only required parameter when configuring a connection. When connecting to the
|
|
||||||
services programmatically, there are different ways in which we can extract and use the rest of AWS configurations.
|
|
||||||
|
|
||||||
You can find further information about configuring your credentials [here](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html#configuring-credentials).
|
|
||||||
|
|
||||||
{% /codeInfo %}
|
|
||||||
|
|
||||||
{% codeInfo srNumber=4 %}
|
|
||||||
|
|
||||||
**endPointURL**: To connect programmatically to an AWS service, you use an endpoint. An *endpoint* is the URL of the
|
|
||||||
entry point for an AWS web service. The AWS SDKs and the AWS Command Line Interface (AWS CLI) automatically use the
|
|
||||||
default endpoint for each service in an AWS Region. But you can specify an alternate endpoint for your API requests.
|
|
||||||
|
|
||||||
Find more information on [AWS service endpoints](https://docs.aws.amazon.com/general/latest/gr/rande.html).
|
|
||||||
|
|
||||||
{% /codeInfo %}
|
|
||||||
|
|
||||||
{% codeInfo srNumber=5 %}
|
|
||||||
|
|
||||||
**profileName**: A named profile is a collection of settings and credentials that you can apply to a AWS CLI command.
|
|
||||||
When you specify a profile to run a command, the settings and credentials are used to run that command.
|
|
||||||
Multiple named profiles can be stored in the config and credentials files.
|
|
||||||
|
|
||||||
You can inform this field if you'd like to use a profile other than `default`.
|
|
||||||
|
|
||||||
Find here more information about [Named profiles for the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-profiles.html).
|
|
||||||
|
|
||||||
{% /codeInfo %}
|
|
||||||
|
|
||||||
{% codeInfo srNumber=6 %}
|
|
||||||
|
|
||||||
**assumeRoleArn**: Typically, you use `AssumeRole` within your account or for cross-account access. In this field you'll set the
|
|
||||||
`ARN` (Amazon Resource Name) of the policy of the other account.
|
|
||||||
|
|
||||||
A user who wants to access a role in a different account must also have permissions that are delegated from the account
|
|
||||||
administrator. The administrator must attach a policy that allows the user to call `AssumeRole` for the `ARN` of the role in the other account.
|
|
||||||
|
|
||||||
This is a required field if you'd like to `AssumeRole`.
|
|
||||||
|
|
||||||
Find more information on [AssumeRole](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html).
|
|
||||||
|
|
||||||
{%note%}
|
|
||||||
When using Assume Role authentication, ensure you provide the following details:
|
|
||||||
- **AWS Region**: Specify the AWS region for your deployment.
|
|
||||||
- **Assume Role ARN**: Provide the ARN of the role in your AWS account that OpenMetadata will assume.
|
|
||||||
{%/note%}
|
|
||||||
|
|
||||||
{% /codeInfo %}
|
|
||||||
|
|
||||||
{% codeInfo srNumber=7 %}
|
|
||||||
|
|
||||||
**assumeRoleSessionName**: An identifier for the assumed role session. Use the role session name to uniquely identify a session when the same role
|
|
||||||
is assumed by different principals or for different reasons.
|
|
||||||
|
|
||||||
By default, we'll use the name `OpenMetadataSession`.
|
|
||||||
|
|
||||||
Find more information about the [Role Session Name](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html#:~:text=An%20identifier%20for%20the%20assumed%20role%20session.).
|
|
||||||
|
|
||||||
{% /codeInfo %}
|
|
||||||
|
|
||||||
{% codeInfo srNumber=8 %}
|
|
||||||
|
|
||||||
**assumeRoleSourceIdentity**: The source identity specified by the principal that is calling the `AssumeRole` operation. You can use source identity
|
|
||||||
information in AWS CloudTrail logs to determine who took actions with a role.
|
|
||||||
|
|
||||||
Find more information about [Source Identity](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html#:~:text=Required%3A%20No-,SourceIdentity,-The%20source%20identity).
|
|
||||||
|
|
||||||
{% /codeInfo %}
|
|
||||||
|
|
||||||
{% partial file="/v1.7/connectors/yaml/storage/source-config-def.md" /%}
|
{% partial file="/v1.7/connectors/yaml/storage/source-config-def.md" /%}
|
||||||
|
|
||||||
@ -262,31 +169,9 @@ source:
|
|||||||
type: S3
|
type: S3
|
||||||
awsConfig:
|
awsConfig:
|
||||||
```
|
```
|
||||||
```yaml {% srNumber=1 %}
|
|
||||||
awsAccessKeyId: KEY
|
{% partial file="/v1.7/connectors/yaml/common/aws-config.md" /%}
|
||||||
awsSecretAccessKey: SECRET
|
|
||||||
```
|
|
||||||
```yaml {% srNumber=2 %}
|
|
||||||
# awsSessionToken: TOKEN
|
|
||||||
```
|
|
||||||
```yaml {% srNumber=3 %}
|
|
||||||
awsRegion: us-east-2
|
|
||||||
```
|
|
||||||
```yaml {% srNumber=4 %}
|
|
||||||
# endPointURL: https://s3.us-east-2.amazonaws.com/custom
|
|
||||||
```
|
|
||||||
```yaml {% srNumber=5 %}
|
|
||||||
# profileName: profile
|
|
||||||
```
|
|
||||||
```yaml {% srNumber=6 %}
|
|
||||||
# assumeRoleArn: "arn:partition:service:region:account:resource"
|
|
||||||
```
|
|
||||||
```yaml {% srNumber=7 %}
|
|
||||||
# assumeRoleSessionName: session
|
|
||||||
```
|
|
||||||
```yaml {% srNumber=8 %}
|
|
||||||
# assumeRoleSourceIdentity: identity
|
|
||||||
```
|
|
||||||
```yaml {% srNumber=13 %}
|
```yaml {% srNumber=13 %}
|
||||||
bucketNames:
|
bucketNames:
|
||||||
- s3-testing-1
|
- s3-testing-1
|
||||||
|
Loading…
x
Reference in New Issue
Block a user