yujunjun/OpenMetadata
1
0
Fork 0
mirror of https://github.com/open-metadata/OpenMetadata.git synced 2025-10-22 14:24:35 +00:00
Code Issues Packages Projects Releases Wiki Activity

Docs: Aws Partial Addition in Connectors (#20342)

Browse Source
This commit is contained in:
Rounak Dhillon 2025-03-24 09:07:16 +05:30 committed by GitHub
parent 6e8fe6ff8f
commit d1ecf60acb
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
24 changed files with 313 additions and 1412 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

94
openmetadata-docs/content/partials/v1.6/connectors/yaml/common/aws-config-def.md Normal file
View File

@ -0,0 +1,94 @@
{% codeInfo srNumber=221 %}
- **awsAccessKeyId** & **awsSecretAccessKey**: When you interact with AWS, you specify your AWS security credentials to verify who you are and whether you have
permission to access the resources that you are requesting. AWS uses the security credentials to authenticate and
authorize your requests ([docs](https://docs.aws.amazon.com/IAM/latest/UserGuide/security-creds.html)).
Access keys consist of two parts: An **access key ID** (for example, `AKIAIOSFODNN7EXAMPLE`), and a **secret access key** (for example, `wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY`).
You must use both the access key ID and secret access key together to authenticate your requests.
You can find further information on how to manage your access keys [here](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html).
{% /codeInfo %}
{% codeInfo srNumber=222 %}
**awsSessionToken**: If you are using temporary credentials to access your services, you will need to inform the AWS Access Key ID
and AWS Secrets Access Key. Also, these will include an AWS Session Token.
{% /codeInfo %}
{% codeInfo srNumber=223 %}
**awsRegion**: Each AWS Region is a separate geographic area in which AWS clusters data centers ([docs](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.RegionsAndAvailabilityZones.html)).
As AWS can have instances in multiple regions, we need to know the region the service you want reach belongs to.
Note that the AWS Region is the only required parameter when configuring a connection. When connecting to the
services programmatically, there are different ways in which we can extract and use the rest of AWS configurations.
You can find further information about configuring your credentials [here](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html#configuring-credentials).
{% /codeInfo %}
{% codeInfo srNumber=224 %}
**endPointURL**: To connect programmatically to an AWS service, you use an endpoint. An *endpoint* is the URL of the
entry point for an AWS web service. The AWS SDKs and the AWS Command Line Interface (AWS CLI) automatically use the
default endpoint for each service in an AWS Region. But you can specify an alternate endpoint for your API requests.
Find more information on [AWS service endpoints](https://docs.aws.amazon.com/general/latest/gr/rande.html).
{% /codeInfo %}
{% codeInfo srNumber=225 %}
**profileName**: A named profile is a collection of settings and credentials that you can apply to a AWS CLI command.
When you specify a profile to run a command, the settings and credentials are used to run that command.
Multiple named profiles can be stored in the config and credentials files.
You can inform this field if you'd like to use a profile other than `default`.
Find here more information about [Named profiles for the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-profiles.html).
{% /codeInfo %}
{% codeInfo srNumber=226 %}
**assumeRoleArn**: Typically, you use `AssumeRole` within your account or for cross-account access. In this field you'll set the
`ARN` (Amazon Resource Name) of the policy of the other account.
A user who wants to access a role in a different account must also have permissions that are delegated from the account
administrator. The administrator must attach a policy that allows the user to call `AssumeRole` for the `ARN` of the role in the other account.
This is a required field if you'd like to `AssumeRole`.
Find more information on [AssumeRole](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html).
{%note%}
When using Assume Role authentication, ensure you provide the following details:
- **AWS Region**: Specify the AWS region for your deployment.
- **Assume Role ARN**: Provide the ARN of the role in your AWS account that OpenMetadata will assume.
{%/note%}
{% /codeInfo %}
{% codeInfo srNumber=227 %}
**assumeRoleSessionName**: An identifier for the assumed role session. Use the role session name to uniquely identify a session when the same role
is assumed by different principals or for different reasons.
By default, we'll use the name `OpenMetadataSession`.
Find more information about the [Role Session Name](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html#:~:text=An%20identifier%20for%20the%20assumed%20role%20session.).
{% /codeInfo %}
{% codeInfo srNumber=228 %}
**assumeRoleSourceIdentity**: The source identity specified by the principal that is calling the `AssumeRole` operation. You can use source identity
information in AWS CloudTrail logs to determine who took actions with a role.
Find more information about [Source Identity](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html#:~:text=Required%3A%20No-,SourceIdentity,-The%20source%20identity).
{% /codeInfo %}

25
openmetadata-docs/content/partials/v1.6/connectors/yaml/common/aws-config.md Normal file
View File

@ -0,0 +1,25 @@
```yaml {% srNumber=221 %}
awsAccessKeyId: KEY
awsSecretAccessKey: SECRET
```
```yaml {% srNumber=222 %}
# awsSessionToken: TOKEN
```
```yaml {% srNumber=223 %}
awsRegion: us-east-2
```
```yaml {% srNumber=224 %}
# endPointURL: https://athena.us-east-2.amazonaws.com/custom
```
```yaml {% srNumber=225 %}
# profileName: profile
```
```yaml {% srNumber=226 %}
# assumeRoleArn: "arn:partition:service:region:account:resource"
```
```yaml {% srNumber=227 %}
# assumeRoleSessionName: session
```
```yaml {% srNumber=228 %}
# assumeRoleSourceIdentity: identity
```

94
openmetadata-docs/content/partials/v1.7/connectors/yaml/common/aws-config-def.md Normal file
View File

@ -0,0 +1,94 @@
{% codeInfo srNumber=221 %}
- **awsAccessKeyId** & **awsSecretAccessKey**: When you interact with AWS, you specify your AWS security credentials to verify who you are and whether you have
permission to access the resources that you are requesting. AWS uses the security credentials to authenticate and
authorize your requests ([docs](https://docs.aws.amazon.com/IAM/latest/UserGuide/security-creds.html)).
Access keys consist of two parts: An **access key ID** (for example, `AKIAIOSFODNN7EXAMPLE`), and a **secret access key** (for example, `wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY`).
You must use both the access key ID and secret access key together to authenticate your requests.
You can find further information on how to manage your access keys [here](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html).
{% /codeInfo %}
{% codeInfo srNumber=222 %}
**awsSessionToken**: If you are using temporary credentials to access your services, you will need to inform the AWS Access Key ID
and AWS Secrets Access Key. Also, these will include an AWS Session Token.
{% /codeInfo %}
{% codeInfo srNumber=223 %}
**awsRegion**: Each AWS Region is a separate geographic area in which AWS clusters data centers ([docs](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.RegionsAndAvailabilityZones.html)).
As AWS can have instances in multiple regions, we need to know the region the service you want reach belongs to.
Note that the AWS Region is the only required parameter when configuring a connection. When connecting to the
services programmatically, there are different ways in which we can extract and use the rest of AWS configurations.
You can find further information about configuring your credentials [here](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html#configuring-credentials).
{% /codeInfo %}
{% codeInfo srNumber=224 %}
**endPointURL**: To connect programmatically to an AWS service, you use an endpoint. An *endpoint* is the URL of the
entry point for an AWS web service. The AWS SDKs and the AWS Command Line Interface (AWS CLI) automatically use the
default endpoint for each service in an AWS Region. But you can specify an alternate endpoint for your API requests.
Find more information on [AWS service endpoints](https://docs.aws.amazon.com/general/latest/gr/rande.html).
{% /codeInfo %}
{% codeInfo srNumber=225 %}
**profileName**: A named profile is a collection of settings and credentials that you can apply to a AWS CLI command.
When you specify a profile to run a command, the settings and credentials are used to run that command.
Multiple named profiles can be stored in the config and credentials files.
You can inform this field if you'd like to use a profile other than `default`.
Find here more information about [Named profiles for the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-profiles.html).
{% /codeInfo %}
{% codeInfo srNumber=226 %}
**assumeRoleArn**: Typically, you use `AssumeRole` within your account or for cross-account access. In this field you'll set the
`ARN` (Amazon Resource Name) of the policy of the other account.
A user who wants to access a role in a different account must also have permissions that are delegated from the account
administrator. The administrator must attach a policy that allows the user to call `AssumeRole` for the `ARN` of the role in the other account.
This is a required field if you'd like to `AssumeRole`.
Find more information on [AssumeRole](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html).
{%note%}
When using Assume Role authentication, ensure you provide the following details:
- **AWS Region**: Specify the AWS region for your deployment.
- **Assume Role ARN**: Provide the ARN of the role in your AWS account that OpenMetadata will assume.
{%/note%}
{% /codeInfo %}
{% codeInfo srNumber=227 %}
**assumeRoleSessionName**: An identifier for the assumed role session. Use the role session name to uniquely identify a session when the same role
is assumed by different principals or for different reasons.
By default, we'll use the name `OpenMetadataSession`.
Find more information about the [Role Session Name](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html#:~:text=An%20identifier%20for%20the%20assumed%20role%20session.).
{% /codeInfo %}
{% codeInfo srNumber=228 %}
**assumeRoleSourceIdentity**: The source identity specified by the principal that is calling the `AssumeRole` operation. You can use source identity
information in AWS CloudTrail logs to determine who took actions with a role.
Find more information about [Source Identity](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html#:~:text=Required%3A%20No-,SourceIdentity,-The%20source%20identity).
{% /codeInfo %}

25
openmetadata-docs/content/partials/v1.7/connectors/yaml/common/aws-config.md Normal file
View File

@ -0,0 +1,25 @@
```yaml {% srNumber=221 %}
awsAccessKeyId: KEY
awsSecretAccessKey: SECRET
```
```yaml {% srNumber=222 %}
# awsSessionToken: TOKEN
```
```yaml {% srNumber=223 %}
awsRegion: us-east-2
```
```yaml {% srNumber=224 %}
# endPointURL: https://athena.us-east-2.amazonaws.com/custom
```
```yaml {% srNumber=225 %}
# profileName: profile
```
```yaml {% srNumber=226 %}
# assumeRoleArn: "arn:partition:service:region:account:resource"
```
```yaml {% srNumber=227 %}
# assumeRoleSessionName: session
```
```yaml {% srNumber=228 %}
# assumeRoleSourceIdentity: identity
```

21
openmetadata-docs/content/v1.6.x/connectors/dashboard/quicksight/yaml.md
View File

@ -103,16 +103,7 @@ This is a sample config for QuickSight:
#### Source Configuration - Service Connection
{% codeInfo srNumber=1 %}
**awsConfig**
- **AWS Access Key ID**: Enter your secure access key ID for your Glue connection. The specified key ID should be authorized to read all databases you want to include in the metadata ingestion workflow.
- **AWS Secret Access Key**: Enter the Secret Access Key (the passcode key pair to the key ID from above).
- **AWS Region**: Enter the location of the amazon cluster that your data and account are associated with.
- **AWS Session Token (optional)**: The AWS session token is an optional parameter. If you want, enter the details of your temporary session token.
- **Endpoint URL (optional)**: Your Glue connector will automatically determine the AWS QuickSight endpoint URL based on the region. You may override this behavior by entering a value to the endpoint URL.
{% /codeInfo %}
{% partial file="/v1.6/connectors/yaml/common/aws-config-def.md" /%}
{% codeInfo srNumber=2 %}
@ -150,13 +141,9 @@ source:
config:
type: QuickSight
```
```yaml {% srNumber=1 %}
awsConfig:
awsAccessKeyId: KEY
awsSecretAccessKey: SECRET
awsRegion: us-east-2
awsSessionToken: Token
```
{% partial file="/v1.6/connectors/yaml/common/aws-config.md" /%}
```yaml {% srNumber=2 %}
awsAccountId: <aws-account-id>
```

122
openmetadata-docs/content/v1.6.x/connectors/database/athena/yaml.md
View File

@ -172,99 +172,7 @@ This is a sample config for Athena:
#### Source Configuration - Service Connection
{% codeInfo srNumber=1 %}
- **awsAccessKeyId** & **awsSecretAccessKey**: When you interact with AWS, you specify your AWS security credentials to verify who you are and whether you have
permission to access the resources that you are requesting. AWS uses the security credentials to authenticate and
authorize your requests ([docs](https://docs.aws.amazon.com/IAM/latest/UserGuide/security-creds.html)).
Access keys consist of two parts: An **access key ID** (for example, `AKIAIOSFODNN7EXAMPLE`), and a **secret access key** (for example, `wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY`).
You must use both the access key ID and secret access key together to authenticate your requests.
You can find further information on how to manage your access keys [here](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html).
{% /codeInfo %}
{% codeInfo srNumber=2 %}
**awsSessionToken**: If you are using temporary credentials to access your services, you will need to inform the AWS Access Key ID
and AWS Secrets Access Key. Also, these will include an AWS Session Token.
{% /codeInfo %}
{% codeInfo srNumber=3 %}
**awsRegion**: Each AWS Region is a separate geographic area in which AWS clusters data centers ([docs](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.RegionsAndAvailabilityZones.html)).
As AWS can have instances in multiple regions, we need to know the region the service you want reach belongs to.
Note that the AWS Region is the only required parameter when configuring a connection. When connecting to the
services programmatically, there are different ways in which we can extract and use the rest of AWS configurations.
You can find further information about configuring your credentials [here](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html#configuring-credentials).
{% /codeInfo %}
{% codeInfo srNumber=4 %}
**endPointURL**: To connect programmatically to an AWS service, you use an endpoint. An *endpoint* is the URL of the
entry point for an AWS web service. The AWS SDKs and the AWS Command Line Interface (AWS CLI) automatically use the
default endpoint for each service in an AWS Region. But you can specify an alternate endpoint for your API requests.
Find more information on [AWS service endpoints](https://docs.aws.amazon.com/general/latest/gr/rande.html).
{% /codeInfo %}
{% codeInfo srNumber=5 %}
**profileName**: A named profile is a collection of settings and credentials that you can apply to a AWS CLI command.
When you specify a profile to run a command, the settings and credentials are used to run that command.
Multiple named profiles can be stored in the config and credentials files.
You can inform this field if you'd like to use a profile other than `default`.
Find here more information about [Named profiles for the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-profiles.html).
{% /codeInfo %}
{% codeInfo srNumber=6 %}
**assumeRoleArn**: Typically, you use `AssumeRole` within your account or for cross-account access. In this field you'll set the
`ARN` (Amazon Resource Name) of the policy of the other account.
A user who wants to access a role in a different account must also have permissions that are delegated from the account
administrator. The administrator must attach a policy that allows the user to call `AssumeRole` for the `ARN` of the role in the other account.
This is a required field if you'd like to `AssumeRole`.
Find more information on [AssumeRole](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html).
{% /codeInfo %}
{%note%}
When using Assume Role authentication, ensure you provide the following details:
- **AWS Region**: Specify the AWS region for your deployment.
- **Assume Role ARN**: Provide the ARN of the role in your AWS account that OpenMetadata will assume.
{%/note%}
{% codeInfo srNumber=7 %}
**assumeRoleSessionName**: An identifier for the assumed role session. Use the role session name to uniquely identify a session when the same role
is assumed by different principals or for different reasons.
By default, we'll use the name `OpenMetadataSession`.
Find more information about the [Role Session Name](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html#:~:text=An%20identifier%20for%20the%20assumed%20role%20session.).
{% /codeInfo %}
{% codeInfo srNumber=8 %}
**assumeRoleSourceIdentity**: The source identity specified by the principal that is calling the `AssumeRole` operation. You can use source identity
information in AWS CloudTrail logs to determine who took actions with a role.
Find more information about [Source Identity](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html#:~:text=Required%3A%20No-,SourceIdentity,-The%20source%20identity).
{% /codeInfo %}
{% partial file="/v1.6/connectors/yaml/common/aws-config-def.md" /%}
{% codeInfo srNumber=9 %}
@ -311,31 +219,9 @@ source:
type: Athena
awsConfig:
```
```yaml {% srNumber=1 %}
awsAccessKeyId: KEY
awsSecretAccessKey: SECRET
```
```yaml {% srNumber=2 %}
# awsSessionToken: TOKEN
```
```yaml {% srNumber=3 %}
awsRegion: us-east-2
```
```yaml {% srNumber=4 %}
# endPointURL: https://athena.us-east-2.amazonaws.com/custom
```
```yaml {% srNumber=5 %}
# profileName: profile
```
```yaml {% srNumber=6 %}
# assumeRoleArn: "arn:partition:service:region:account:resource"
```
```yaml {% srNumber=7 %}
# assumeRoleSessionName: session
```
```yaml {% srNumber=8 %}
# assumeRoleSourceIdentity: identity
```
{% partial file="/v1.6/connectors/yaml/common/aws-config.md" /%}
```yaml {% srNumber=9 %}
s3StagingDir: s3 directory for datasource
```

48
openmetadata-docs/content/v1.6.x/connectors/database/dynamodb/yaml.md
View File

@ -78,35 +78,7 @@ This is a sample config for DynamoDB:
#### Source Configuration - Service Connection
{% codeInfo srNumber=1 %}
**awsAccessKeyId**: Enter your secure access key ID for your DynamoDB connection. The specified key ID should be authorized to read all databases you want to include in the metadata ingestion workflow.
{% /codeInfo %}
{% codeInfo srNumber=2 %}
**awsSecretAccessKey**: Enter the Secret Access Key (the passcode key pair to the key ID from above).
{% /codeInfo %}
{% codeInfo srNumber=3 %}
**awsSessionToken**: The AWS session token is an optional parameter. If you want, enter the details of your temporary session token.
{% /codeInfo %}
{% codeInfo srNumber=4 %}
**awsRegion**: Enter the location of the amazon cluster that your data and account are associated with.
{% /codeInfo %}
{% codeInfo srNumber=5 %}
**endPointURL**: Your DynamoDB connector will automatically determine the AWS DynamoDB endpoint URL based on the region. You may override this behavior by entering a value to the endpoint URL.
{% /codeInfo %}
{% partial file="/v1.6/connectors/yaml/common/aws-config-def.md" /%}
{% codeInfo srNumber=6 %}
@ -149,21 +121,9 @@ source:
type: DynamoDB
awsConfig:
```
```yaml {% srNumber=1 %}
awsAccessKeyId: aws_access_key_id
```
```yaml {% srNumber=2 %}
awsSecretAccessKey: aws_secret_access_key
```
```yaml {% srNumber=3 %}
awsSessionToken: AWS Session Token
```
```yaml {% srNumber=4 %}
awsRegion: aws region
```
```yaml {% srNumber=5 %}
endPointURL: https://dynamodb.<region_name>.amazonaws.com
```
{% partial file="/v1.6/connectors/yaml/common/aws-config.md" /%}
```yaml {% srNumber=6 %}
database: custom_database_name
```

48
openmetadata-docs/content/v1.6.x/connectors/database/glue/yaml.md
View File

@ -58,35 +58,7 @@ This is a sample config for Glue:
#### Source Configuration - Service Connection
{% codeInfo srNumber=1 %}
**awsAccessKeyId**: Enter your secure access key ID for your Glue connection. The specified key ID should be authorized to read all databases you want to include in the metadata ingestion workflow.
{% /codeInfo %}
{% codeInfo srNumber=2 %}
**awsSecretAccessKey**: Enter the Secret Access Key (the passcode key pair to the key ID from above).
{% /codeInfo %}
{% codeInfo srNumber=3 %}
**awsRegion**: Enter the location of the amazon cluster that your data and account are associated with.
{% /codeInfo %}
{% codeInfo srNumber=4 %}
**endPointURL**: Your Glue connector will automatically determine the AWS Glue endpoint URL based on the region. You may override this behavior by entering a value to the endpoint URL.
{% /codeInfo %}
{% codeInfo srNumber=5 %}
**awsSessionToken**: The AWS session token is an optional parameter. If you want, enter the details of your temporary session token.
{% /codeInfo %}
{% partial file="/v1.6/connectors/yaml/common/aws-config-def.md" /%}
{% codeInfo srNumber=6 %}
@ -130,21 +102,9 @@ source:
type: Glue
awsConfig:
```
```yaml {% srNumber=1 %}
awsAccessKeyId: KEY
```
```yaml {% srNumber=2 %}
awsSecretAccessKey: SECRET
```
```yaml {% srNumber=3 %}
awsRegion: us-east-2
```
```yaml {% srNumber=4 %}
# endPointURL: https://glue.us-east-2.amazonaws.com/
```
```yaml {% srNumber=5 %}
# awsSessionToken: TOKEN
```
{% partial file="/v1.6/connectors/yaml/common/aws-config.md" /%}
```yaml {% srNumber=6 %}
databaseName: database_name
```

12
openmetadata-docs/content/v1.6.x/connectors/database/s3-datalake/yaml.md
View File

@ -90,13 +90,11 @@ The workflow is modeled around the following JSON Schema.
{% codeInfo srNumber=1 %}
* **awsAccessKeyId**: Enter your secure access key ID for your DynamoDB connection. The specified key ID should be authorized to read all databases you want to include in the metadata ingestion workflow.
* **awsSecretAccessKey**: Enter the Secret Access Key (the passcode key pair to the key ID from above).
* **awsRegion**: Specify the region in which your DynamoDB is located. This setting is required even if you have configured a local AWS profile.
* **schemaFilterPattern** and **tableFilterPattern**: Note that the `schemaFilterPattern` and `tableFilterPattern` both support regex as `include` or `exclude`. E.g.,
{% /codeInfo %}
{% partial file="/v1.6/connectors/yaml/common/aws-config-def.md" /%}
{% partial file="/v1.6/connectors/yaml/database/source-config-def.md" /%}
@ -120,9 +118,11 @@ source:
```yaml {% srNumber=1 %}
configSource:
securityConfig:
awsAccessKeyId: aws access key id
awsSecretAccessKey: aws secret access key
awsRegion: aws region
```
{% partial file="/v1.6/connectors/yaml/common/aws-config.md" /%}
```yaml {% srNumber=1 %}
bucketName: bucket name
prefix: prefix
```

43
openmetadata-docs/content/v1.6.x/connectors/ingestion/workflows/dbt/ingest-dbt-yaml.md
View File

@ -46,38 +46,7 @@ In this configuration we will be fetching the dbt `manifest.json`, `catalog.json
- **dbtConfigType**: s3
{% /codeInfo %}
{% codeInfo srNumber=1 %}
- **awsAccessKeyId** & **awsSecretAccessKey**: When you interact with AWS, you specify your AWS security credentials to verify who you are and whether you have
permission to access the resources that you are requesting. AWS uses the security credentials to authenticate and
authorize your requests ([docs](https://docs.aws.amazon.com/IAM/latest/UserGuide/security-creds.html)).
Access keys consist of two parts: An **access key ID** (for example, `AKIAIOSFODNN7EXAMPLE`), and a **secret access key** (for example, `wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY`).
You must use both the access key ID and secret access key together to authenticate your requests.
You can find further information on how to manage your access keys [here](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html).
{% /codeInfo %}
{% codeInfo srNumber=2 %}
**awsSessionToken**: If you are using temporary credentials to access your services, you will need to inform the AWS Access Key ID
and AWS Secrets Access Key. Also, these will include an AWS Session Token.
{% /codeInfo %}
{% codeInfo srNumber=3 %}
**awsRegion**: Each AWS Region is a separate geographic area in which AWS clusters data centers ([docs](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.RegionsAndAvailabilityZones.html)).
As AWS can have instances in multiple regions, we need to know the region the service you want reach belongs to.
Note that the AWS Region is the only required parameter when configuring a connection. When connecting to the
services programmatically, there are different ways in which we can extract and use the rest of AWS configurations.
You can find further information about configuring your credentials [here](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html#configuring-credentials).
{% /codeInfo %}
{% partial file="/v1.6/connectors/yaml/common/aws-config-def.md" /%}
{% partial file="/v1.6/connectors/yaml/dbt/dbt-prefix-def.md" /%}
@ -105,16 +74,10 @@ source:
```
```yaml {% srNumber=1 %}
dbtSecurityConfig:
awsAccessKeyId: AWS_ACCESS_KEY_ID
awsSecretAccessKey: AWS_SECRET_ACCESS_KEY
```
```yaml {% srNumber=2 %}
awsSessionToken: AWS_SESSION_TOKEN
```
```yaml {% srNumber=3 %}
awsRegion: AWS_REGION
```
{% partial file="/v1.6/connectors/yaml/common/aws-config.md" /%}
{% partial file="/v1.6/connectors/yaml/dbt/dbt-prefix.md" /%}
{% partial file="/v1.6/connectors/yaml/dbt/source-config.md" /%}

123
openmetadata-docs/content/v1.6.x/connectors/messaging/kinesis/yaml.md
View File

@ -80,100 +80,7 @@ This is a sample config for Kinesis:
#### Source Configuration - Service Connection
{% codeInfo srNumber=1 %}
- **awsAccessKeyId** & **awsSecretAccessKey**: When you interact with AWS, you specify your AWS security credentials to verify who you are and whether you have
permission to access the resources that you are requesting. AWS uses the security credentials to authenticate and
authorize your requests ([docs](https://docs.aws.amazon.com/IAM/latest/UserGuide/security-creds.html)).
Access keys consist of two parts: An **access key ID** (for example, `AKIAIOSFODNN7EXAMPLE`), and a **secret access key** (for example, `wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY`).
You must use both the access key ID and secret access key together to authenticate your requests.
You can find further information on how to manage your access keys [here](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html).
{% /codeInfo %}
{% codeInfo srNumber=2 %}
**awsSessionToken**: If you are using temporary credentials to access your services, you will need to inform the AWS Access Key ID
and AWS Secrets Access Key. Also, these will include an AWS Session Token.
{% /codeInfo %}
{% codeInfo srNumber=3 %}
**awsRegion**: Each AWS Region is a separate geographic area in which AWS clusters data centers ([docs](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.RegionsAndAvailabilityZones.html)).
As AWS can have instances in multiple regions, we need to know the region the service you want reach belongs to.
Note that the AWS Region is the only required parameter when configuring a connection. When connecting to the
services programmatically, there are different ways in which we can extract and use the rest of AWS configurations.
You can find further information about configuring your credentials [here](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html#configuring-credentials).
{% /codeInfo %}
{% codeInfo srNumber=4 %}
**endPointURL**: To connect programmatically to an AWS service, you use an endpoint. An *endpoint* is the URL of the
entry point for an AWS web service. The AWS SDKs and the AWS Command Line Interface (AWS CLI) automatically use the
default endpoint for each service in an AWS Region. But you can specify an alternate endpoint for your API requests.
Find more information on [AWS service endpoints](https://docs.aws.amazon.com/general/latest/gr/rande.html).
{% /codeInfo %}
{% codeInfo srNumber=5 %}
**profileName**: A named profile is a collection of settings and credentials that you can apply to a AWS CLI command.
When you specify a profile to run a command, the settings and credentials are used to run that command.
Multiple named profiles can be stored in the config and credentials files.
You can inform this field if you'd like to use a profile other than `default`.
Find here more information about [Named profiles for the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-profiles.html).
{% /codeInfo %}
{% codeInfo srNumber=6 %}
**assumeRoleArn**: Typically, you use `AssumeRole` within your account or for cross-account access. In this field you'll set the
`ARN` (Amazon Resource Name) of the policy of the other account.
A user who wants to access a role in a different account must also have permissions that are delegated from the account
administrator. The administrator must attach a policy that allows the user to call `AssumeRole` for the `ARN` of the role in the other account.
This is a required field if you'd like to `AssumeRole`.
Find more information on [AssumeRole](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html).
{%note%}
When using Assume Role authentication, ensure you provide the following details:
- **AWS Region**: Specify the AWS region for your deployment.
- **Assume Role ARN**: Provide the ARN of the role in your AWS account that OpenMetadata will assume.
{%/note%}
{% /codeInfo %}
{% codeInfo srNumber=7 %}
**assumeRoleSessionName**: An identifier for the assumed role session. Use the role session name to uniquely identify a session when the same role
is assumed by different principals or for different reasons.
By default, we'll use the name `OpenMetadataSession`.
Find more information about the [Role Session Name](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html#:~:text=An%20identifier%20for%20the%20assumed%20role%20session.).
{% /codeInfo %}
{% codeInfo srNumber=8 %}
**assumeRoleSourceIdentity**: The source identity specified by the principal that is calling the `AssumeRole` operation. You can use source identity
information in AWS CloudTrail logs to determine who took actions with a role.
Find more information about [Source Identity](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html#:~:text=Required%3A%20No-,SourceIdentity,-The%20source%20identity).
{% /codeInfo %}
{% partial file="/v1.6/connectors/yaml/common/aws-config-def.md" /%}
{% partial file="/v1.6/connectors/yaml/messaging/source-config-def.md" /%}
@ -195,31 +102,9 @@ source:
type: Kinesis
awsConfig:
```
```yaml {% srNumber=1 %}
awsAccessKeyId: KEY
awsSecretAccessKey: SECRET
```
```yaml {% srNumber=2 %}
# awsSessionToken: TOKEN
```
```yaml {% srNumber=3 %}
awsRegion: us-east-2
```
```yaml {% srNumber=4 %}
# endPointURL: https://athena.us-east-2.amazonaws.com/custom
```
```yaml {% srNumber=5 %}
# profileName: profile
```
```yaml {% srNumber=6 %}
# assumeRoleArn: "arn:partition:service:region:account:resource"
```
```yaml {% srNumber=7 %}
# assumeRoleSessionName: session
```
```yaml {% srNumber=8 %}
# assumeRoleSourceIdentity: identity
```
{% partial file="/v1.6/connectors/yaml/common/aws-config.md" /%}
{% partial file="/v1.6/connectors/yaml/messaging/source-config.md" /%}

122
openmetadata-docs/content/v1.6.x/connectors/ml-model/sagemaker/yaml.md
View File

@ -78,100 +78,7 @@ This is a sample config for Sagemaker:
#### Source Configuration - Service Connection
{% codeInfo srNumber=1 %}
- **awsAccessKeyId** & **awsSecretAccessKey**: When you interact with AWS, you specify your AWS security credentials to verify who you are and whether you have
permission to access the resources that you are requesting. AWS uses the security credentials to authenticate and
authorize your requests ([docs](https://docs.aws.amazon.com/IAM/latest/UserGuide/security-creds.html)).
Access keys consist of two parts: An **access key ID** (for example, `AKIAIOSFODNN7EXAMPLE`), and a **secret access key** (for example, `wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY`).
You must use both the access key ID and secret access key together to authenticate your requests.
You can find further information on how to manage your access keys [here](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html).
{% /codeInfo %}
{% codeInfo srNumber=2 %}
**awsSessionToken**: If you are using temporary credentials to access your services, you will need to inform the AWS Access Key ID
and AWS Secrets Access Key. Also, these will include an AWS Session Token.
{% /codeInfo %}
{% codeInfo srNumber=3 %}
**awsRegion**: Each AWS Region is a separate geographic area in which AWS clusters data centers ([docs](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.RegionsAndAvailabilityZones.html)).
As AWS can have instances in multiple regions, we need to know the region the service you want reach belongs to.
Note that the AWS Region is the only required parameter when configuring a connection. When connecting to the
services programmatically, there are different ways in which we can extract and use the rest of AWS configurations.
You can find further information about configuring your credentials [here](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html#configuring-credentials).
{% /codeInfo %}
{% codeInfo srNumber=4 %}
**endPointURL**: To connect programmatically to an AWS service, you use an endpoint. An *endpoint* is the URL of the
entry point for an AWS web service. The AWS SDKs and the AWS Command Line Interface (AWS CLI) automatically use the
default endpoint for each service in an AWS Region. But you can specify an alternate endpoint for your API requests.
Find more information on [AWS service endpoints](https://docs.aws.amazon.com/general/latest/gr/rande.html).
{% /codeInfo %}
{% codeInfo srNumber=5 %}
**profileName**: A named profile is a collection of settings and credentials that you can apply to a AWS CLI command.
When you specify a profile to run a command, the settings and credentials are used to run that command.
Multiple named profiles can be stored in the config and credentials files.
You can inform this field if you'd like to use a profile other than `default`.
Find here more information about [Named profiles for the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-profiles.html).
{% /codeInfo %}
{% codeInfo srNumber=6 %}
**assumeRoleArn**: Typically, you use `AssumeRole` within your account or for cross-account access. In this field you'll set the
`ARN` (Amazon Resource Name) of the policy of the other account.
A user who wants to access a role in a different account must also have permissions that are delegated from the account
administrator. The administrator must attach a policy that allows the user to call `AssumeRole` for the `ARN` of the role in the other account.
This is a required field if you'd like to `AssumeRole`.
Find more information on [AssumeRole](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html).
{%note%}
When using Assume Role authentication, ensure you provide the following details:
- **AWS Region**: Specify the AWS region for your deployment.
- **Assume Role ARN**: Provide the ARN of the role in your AWS account that OpenMetadata will assume.
{%/note%}
{% /codeInfo %}
{% codeInfo srNumber=7 %}
**assumeRoleSessionName**: An identifier for the assumed role session. Use the role session name to uniquely identify a session when the same role
is assumed by different principals or for different reasons.
By default, we'll use the name `OpenMetadataSession`.
Find more information about the [Role Session Name](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html#:~:text=An%20identifier%20for%20the%20assumed%20role%20session.).
{% /codeInfo %}
{% codeInfo srNumber=8 %}
**assumeRoleSourceIdentity**: The source identity specified by the principal that is calling the `AssumeRole` operation. You can use source identity
information in AWS CloudTrail logs to determine who took actions with a role.
Find more information about [Source Identity](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html#:~:text=Required%3A%20No-,SourceIdentity,-The%20source%20identity).
{% /codeInfo %}
{% partial file="/v1.6/connectors/yaml/common/aws-config-def.md" /%}
{% partial file="/v1.6/connectors/yaml/ml-model/source-config-def.md" /%}
@ -193,31 +100,8 @@ source:
type: SageMaker
awsConfig:
```
```yaml {% srNumber=1 %}
awsAccessKeyId: KEY
awsSecretAccessKey: SECRET
```
```yaml {% srNumber=2 %}
# awsSessionToken: TOKEN
```
```yaml {% srNumber=3 %}
awsRegion: us-east-2
```
```yaml {% srNumber=4 %}
# endPointURL: https://athena.us-east-2.amazonaws.com/custom
```
```yaml {% srNumber=5 %}
# profileName: profile
```
```yaml {% srNumber=6 %}
# assumeRoleArn: "arn:partition:service:region:account:resource"
```
```yaml {% srNumber=7 %}
# assumeRoleSessionName: session
```
```yaml {% srNumber=8 %}
# assumeRoleSourceIdentity: identity
```
{% partial file="/v1.6/connectors/yaml/common/aws-config.md" /%}
{% partial file="/v1.6/connectors/yaml/ml-model/source-config.md" /%}

81
openmetadata-docs/content/v1.6.x/connectors/pipeline/glue-pipeline/yaml.md
View File

@ -63,58 +63,7 @@ This is a sample config for Glue:
#### Source Configuration - Service Connection
{% codeInfo srNumber=1 %}
**awsAccessKeyId**: Enter your secure access key ID for your Glue connection. The specified key ID should be
authorized to read all databases you want to include in the metadata ingestion workflow.
{% /codeInfo %}
{% codeInfo srNumber=2 %}
**awsSecretAccessKey**: Enter the Secret Access Key (the passcode key pair to the key ID from above).
{% /codeInfo %}
{% codeInfo srNumber=3 %}
**awsRegion**: Enter the location of the amazon cluster that your data and account are associated with.
{% /codeInfo %}
{% codeInfo srNumber=4 %}
**awsSessionToken**: The AWS session token is an optional parameter. If you want, enter the details of your temporary
session token.
{% /codeInfo %}
{% codeInfo srNumber=5 %}
**endPointURL**: Your Glue connector will automatically determine the AWS Glue endpoint URL based on the region. You
may override this behavior by entering a value to the endpoint URL.
{% /codeInfo %}
{% codeInfo srNumber=6 %}
**profileName**: The name of a profile to use with the boto session.
{% /codeInfo %}
{% codeInfo srNumber=7 %}
**assumeRoleArn**: The Amazon Resource Name (ARN) of the role to assume. Required Field in case of Assume Role.
{% /codeInfo %}
{% codeInfo srNumber=8 %}
**assumeRoleSessionName**: An identifier for the assumed role session. Use the role session name to uniquely identify a session when the same role is assumed by different principals or for different reasons. Required Field in case of Assume Role.
{% /codeInfo %}
{% codeInfo srNumber=9 %}
**assumeRoleSourceIdentity**: The Amazon Resource Name (ARN) of the role to assume. Optional Field in case of Assume Role.
{% /codeInfo %}
{% partial file="/v1.6/connectors/yaml/common/aws-config-def.md" /%}
{% partial file="/v1.6/connectors/yaml/pipeline/source-config-def.md" /%}
@ -135,34 +84,8 @@ source:
type: GluePipeline
awsConfig:
```
```yaml {% srNumber=1 %}
awsAccessKeyId: KEY
```
```yaml {% srNumber=2 %}
awsSecretAccessKey: SECRET
```
```yaml {% srNumber=3 %}
awsRegion: us-east-2
```
```yaml {% srNumber=4 %}
# awsSessionToken: TOKEN
```
```yaml {% srNumber=5 %}
# endPointURL: https://glue.us-east-2.amazonaws.com/
```
```yaml {% srNumber=6 %}
# profileName: ""
```
```yaml {% srNumber=7 %}
# assumeRoleArn: ""
```
```yaml {% srNumber=8 %}
# assumeRoleSessionName: OpenMetadataSession
```
```yaml {% srNumber=9 %}
# assumeRoleSourceIdentity : ""
```
{% partial file="/v1.6/connectors/yaml/common/aws-config.md" /%}
{% partial file="/v1.6/connectors/yaml/pipeline/source-config.md" /%}

123
openmetadata-docs/content/v1.6.x/connectors/storage/s3/yaml.md
View File

@ -128,100 +128,7 @@ This is a sample config for Athena:
#### Source Configuration - Service Connection
{% codeInfo srNumber=1 %}
- **awsAccessKeyId** & **awsSecretAccessKey**: When you interact with AWS, you specify your AWS security credentials to verify who you are and whether you have
permission to access the resources that you are requesting. AWS uses the security credentials to authenticate and
authorize your requests ([docs](https://docs.aws.amazon.com/IAM/latest/UserGuide/security-creds.html)).
Access keys consist of two parts: An **access key ID** (for example, `AKIAIOSFODNN7EXAMPLE`), and a **secret access key** (for example, `wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY`).
You must use both the access key ID and secret access key together to authenticate your requests.
You can find further information on how to manage your access keys [here](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html).
{% /codeInfo %}
{% codeInfo srNumber=2 %}
**awsSessionToken**: If you are using temporary credentials to access your services, you will need to inform the AWS Access Key ID
and AWS Secrets Access Key. Also, these will include an AWS Session Token.
{% /codeInfo %}
{% codeInfo srNumber=3 %}
**awsRegion**: Each AWS Region is a separate geographic area in which AWS clusters data centers ([docs](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.RegionsAndAvailabilityZones.html)).
As AWS can have instances in multiple regions, we need to know the region the service you want reach belongs to.
Note that the AWS Region is the only required parameter when configuring a connection. When connecting to the
services programmatically, there are different ways in which we can extract and use the rest of AWS configurations.
You can find further information about configuring your credentials [here](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html#configuring-credentials).
{% /codeInfo %}
{% codeInfo srNumber=4 %}
**endPointURL**: To connect programmatically to an AWS service, you use an endpoint. An *endpoint* is the URL of the
entry point for an AWS web service. The AWS SDKs and the AWS Command Line Interface (AWS CLI) automatically use the
default endpoint for each service in an AWS Region. But you can specify an alternate endpoint for your API requests.
Find more information on [AWS service endpoints](https://docs.aws.amazon.com/general/latest/gr/rande.html).
{% /codeInfo %}
{% codeInfo srNumber=5 %}
**profileName**: A named profile is a collection of settings and credentials that you can apply to a AWS CLI command.
When you specify a profile to run a command, the settings and credentials are used to run that command.
Multiple named profiles can be stored in the config and credentials files.
You can inform this field if you'd like to use a profile other than `default`.
Find here more information about [Named profiles for the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-profiles.html).
{% /codeInfo %}
{% codeInfo srNumber=6 %}
**assumeRoleArn**: Typically, you use `AssumeRole` within your account or for cross-account access. In this field you'll set the
`ARN` (Amazon Resource Name) of the policy of the other account.
A user who wants to access a role in a different account must also have permissions that are delegated from the account
administrator. The administrator must attach a policy that allows the user to call `AssumeRole` for the `ARN` of the role in the other account.
This is a required field if you'd like to `AssumeRole`.
Find more information on [AssumeRole](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html).
{%note%}
When using Assume Role authentication, ensure you provide the following details:
- **AWS Region**: Specify the AWS region for your deployment.
- **Assume Role ARN**: Provide the ARN of the role in your AWS account that OpenMetadata will assume.
{%/note%}
{% /codeInfo %}
{% codeInfo srNumber=7 %}
**assumeRoleSessionName**: An identifier for the assumed role session. Use the role session name to uniquely identify a session when the same role
is assumed by different principals or for different reasons.
By default, we'll use the name `OpenMetadataSession`.
Find more information about the [Role Session Name](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html#:~:text=An%20identifier%20for%20the%20assumed%20role%20session.).
{% /codeInfo %}
{% codeInfo srNumber=8 %}
**assumeRoleSourceIdentity**: The source identity specified by the principal that is calling the `AssumeRole` operation. You can use source identity
information in AWS CloudTrail logs to determine who took actions with a role.
Find more information about [Source Identity](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html#:~:text=Required%3A%20No-,SourceIdentity,-The%20source%20identity).
{% /codeInfo %}
{% partial file="/v1.6/connectors/yaml/common/aws-config-def.md" /%}
{% partial file="/v1.6/connectors/yaml/storage/source-config-def.md" /%}
@ -262,31 +169,9 @@ source:
type: S3
awsConfig:
```
```yaml {% srNumber=1 %}
awsAccessKeyId: KEY
awsSecretAccessKey: SECRET
```
```yaml {% srNumber=2 %}
# awsSessionToken: TOKEN
```
```yaml {% srNumber=3 %}
awsRegion: us-east-2
```
```yaml {% srNumber=4 %}
# endPointURL: https://s3.us-east-2.amazonaws.com/custom
```
```yaml {% srNumber=5 %}
# profileName: profile
```
```yaml {% srNumber=6 %}
# assumeRoleArn: "arn:partition:service:region:account:resource"
```
```yaml {% srNumber=7 %}
# assumeRoleSessionName: session
```
```yaml {% srNumber=8 %}
# assumeRoleSourceIdentity: identity
```
{% partial file="/v1.6/connectors/yaml/common/aws-config.md" /%}
```yaml {% srNumber=13 %}
bucketNames:
- s3-testing-1

21
openmetadata-docs/content/v1.7.x-SNAPSHOT/connectors/dashboard/quicksight/yaml.md
View File

@ -103,16 +103,7 @@ This is a sample config for QuickSight:
#### Source Configuration - Service Connection
{% codeInfo srNumber=1 %}
**awsConfig**
- **AWS Access Key ID**: Enter your secure access key ID for your Glue connection. The specified key ID should be authorized to read all databases you want to include in the metadata ingestion workflow.
- **AWS Secret Access Key**: Enter the Secret Access Key (the passcode key pair to the key ID from above).
- **AWS Region**: Enter the location of the amazon cluster that your data and account are associated with.
- **AWS Session Token (optional)**: The AWS session token is an optional parameter. If you want, enter the details of your temporary session token.
- **Endpoint URL (optional)**: Your Glue connector will automatically determine the AWS QuickSight endpoint URL based on the region. You may override this behavior by entering a value to the endpoint URL.
{% /codeInfo %}
{% partial file="/v1.7/connectors/yaml/common/aws-config-def.md" /%}
{% codeInfo srNumber=2 %}
@ -150,13 +141,9 @@ source:
config:
type: QuickSight
```
```yaml {% srNumber=1 %}
awsConfig:
awsAccessKeyId: KEY
awsSecretAccessKey: SECRET
awsRegion: us-east-2
awsSessionToken: Token
```
{% partial file="/v1.7/connectors/yaml/common/aws-config.md" /%}
```yaml {% srNumber=2 %}
awsAccountId: <aws-account-id>
```

123
openmetadata-docs/content/v1.7.x-SNAPSHOT/connectors/database/athena/yaml.md
View File

@ -172,100 +172,7 @@ This is a sample config for Athena:
#### Source Configuration - Service Connection
{% codeInfo srNumber=1 %}
- **awsAccessKeyId** & **awsSecretAccessKey**: When you interact with AWS, you specify your AWS security credentials to verify who you are and whether you have
permission to access the resources that you are requesting. AWS uses the security credentials to authenticate and
authorize your requests ([docs](https://docs.aws.amazon.com/IAM/latest/UserGuide/security-creds.html)).
Access keys consist of two parts: An **access key ID** (for example, `AKIAIOSFODNN7EXAMPLE`), and a **secret access key** (for example, `wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY`).
You must use both the access key ID and secret access key together to authenticate your requests.
You can find further information on how to manage your access keys [here](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html).
{% /codeInfo %}
{% codeInfo srNumber=2 %}
**awsSessionToken**: If you are using temporary credentials to access your services, you will need to inform the AWS Access Key ID
and AWS Secrets Access Key. Also, these will include an AWS Session Token.
{% /codeInfo %}
{% codeInfo srNumber=3 %}
**awsRegion**: Each AWS Region is a separate geographic area in which AWS clusters data centers ([docs](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.RegionsAndAvailabilityZones.html)).
As AWS can have instances in multiple regions, we need to know the region the service you want reach belongs to.
Note that the AWS Region is the only required parameter when configuring a connection. When connecting to the
services programmatically, there are different ways in which we can extract and use the rest of AWS configurations.
You can find further information about configuring your credentials [here](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html#configuring-credentials).
{% /codeInfo %}
{% codeInfo srNumber=4 %}
**endPointURL**: To connect programmatically to an AWS service, you use an endpoint. An *endpoint* is the URL of the
entry point for an AWS web service. The AWS SDKs and the AWS Command Line Interface (AWS CLI) automatically use the
default endpoint for each service in an AWS Region. But you can specify an alternate endpoint for your API requests.
Find more information on [AWS service endpoints](https://docs.aws.amazon.com/general/latest/gr/rande.html).
{% /codeInfo %}
{% codeInfo srNumber=5 %}
**profileName**: A named profile is a collection of settings and credentials that you can apply to a AWS CLI command.
When you specify a profile to run a command, the settings and credentials are used to run that command.
Multiple named profiles can be stored in the config and credentials files.
You can inform this field if you'd like to use a profile other than `default`.
Find here more information about [Named profiles for the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-profiles.html).
{% /codeInfo %}
{% codeInfo srNumber=6 %}
**assumeRoleArn**: Typically, you use `AssumeRole` within your account or for cross-account access. In this field you'll set the
`ARN` (Amazon Resource Name) of the policy of the other account.
A user who wants to access a role in a different account must also have permissions that are delegated from the account
administrator. The administrator must attach a policy that allows the user to call `AssumeRole` for the `ARN` of the role in the other account.
This is a required field if you'd like to `AssumeRole`.
Find more information on [AssumeRole](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html).
{%note%}
When using Assume Role authentication, ensure you provide the following details:
- **AWS Region**: Specify the AWS region for your deployment.
- **Assume Role ARN**: Provide the ARN of the role in your AWS account that OpenMetadata will assume.
{%/note%}
{% /codeInfo %}
{% codeInfo srNumber=7 %}
**assumeRoleSessionName**: An identifier for the assumed role session. Use the role session name to uniquely identify a session when the same role
is assumed by different principals or for different reasons.
By default, we'll use the name `OpenMetadataSession`.
Find more information about the [Role Session Name](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html#:~:text=An%20identifier%20for%20the%20assumed%20role%20session.).
{% /codeInfo %}
{% codeInfo srNumber=8 %}
**assumeRoleSourceIdentity**: The source identity specified by the principal that is calling the `AssumeRole` operation. You can use source identity
information in AWS CloudTrail logs to determine who took actions with a role.
Find more information about [Source Identity](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html#:~:text=Required%3A%20No-,SourceIdentity,-The%20source%20identity).
{% /codeInfo %}
{% partial file="/v1.7/connectors/yaml/common/aws-config-def.md" /%}
{% codeInfo srNumber=9 %}
@ -312,31 +219,9 @@ source:
type: Athena
awsConfig:
```
```yaml {% srNumber=1 %}
awsAccessKeyId: KEY
awsSecretAccessKey: SECRET
```
```yaml {% srNumber=2 %}
# awsSessionToken: TOKEN
```
```yaml {% srNumber=3 %}
awsRegion: us-east-2
```
```yaml {% srNumber=4 %}
# endPointURL: https://athena.us-east-2.amazonaws.com/custom
```
```yaml {% srNumber=5 %}
# profileName: profile
```
```yaml {% srNumber=6 %}
# assumeRoleArn: "arn:partition:service:region:account:resource"
```
```yaml {% srNumber=7 %}
# assumeRoleSessionName: session
```
```yaml {% srNumber=8 %}
# assumeRoleSourceIdentity: identity
```
{% partial file="/v1.7/connectors/yaml/common/aws-config.md" /%}
```yaml {% srNumber=9 %}
s3StagingDir: s3 directory for datasource
```

48
openmetadata-docs/content/v1.7.x-SNAPSHOT/connectors/database/dynamodb/yaml.md
View File

@ -78,35 +78,7 @@ This is a sample config for DynamoDB:
#### Source Configuration - Service Connection
{% codeInfo srNumber=1 %}
**awsAccessKeyId**: Enter your secure access key ID for your DynamoDB connection. The specified key ID should be authorized to read all databases you want to include in the metadata ingestion workflow.
{% /codeInfo %}
{% codeInfo srNumber=2 %}
**awsSecretAccessKey**: Enter the Secret Access Key (the passcode key pair to the key ID from above).
{% /codeInfo %}
{% codeInfo srNumber=3 %}
**awsSessionToken**: The AWS session token is an optional parameter. If you want, enter the details of your temporary session token.
{% /codeInfo %}
{% codeInfo srNumber=4 %}
**awsRegion**: Enter the location of the amazon cluster that your data and account are associated with.
{% /codeInfo %}
{% codeInfo srNumber=5 %}
**endPointURL**: Your DynamoDB connector will automatically determine the AWS DynamoDB endpoint URL based on the region. You may override this behavior by entering a value to the endpoint URL.
{% /codeInfo %}
{% partial file="/v1.7/connectors/yaml/common/aws-config-def.md" /%}
{% codeInfo srNumber=6 %}
@ -149,21 +121,9 @@ source:
type: DynamoDB
awsConfig:
```
```yaml {% srNumber=1 %}
awsAccessKeyId: aws_access_key_id
```
```yaml {% srNumber=2 %}
awsSecretAccessKey: aws_secret_access_key
```
```yaml {% srNumber=3 %}
awsSessionToken: AWS Session Token
```
```yaml {% srNumber=4 %}
awsRegion: aws region
```
```yaml {% srNumber=5 %}
endPointURL: https://dynamodb.<region_name>.amazonaws.com
```
{% partial file="/v1.7/connectors/yaml/common/aws-config.md" /%}
```yaml {% srNumber=6 %}
database: custom_database_name
```

48
openmetadata-docs/content/v1.7.x-SNAPSHOT/connectors/database/glue/yaml.md
View File

@ -58,35 +58,7 @@ This is a sample config for Glue:
#### Source Configuration - Service Connection
{% codeInfo srNumber=1 %}
**awsAccessKeyId**: Enter your secure access key ID for your Glue connection. The specified key ID should be authorized to read all databases you want to include in the metadata ingestion workflow.
{% /codeInfo %}
{% codeInfo srNumber=2 %}
**awsSecretAccessKey**: Enter the Secret Access Key (the passcode key pair to the key ID from above).
{% /codeInfo %}
{% codeInfo srNumber=3 %}
**awsRegion**: Enter the location of the amazon cluster that your data and account are associated with.
{% /codeInfo %}
{% codeInfo srNumber=4 %}
**endPointURL**: Your Glue connector will automatically determine the AWS Glue endpoint URL based on the region. You may override this behavior by entering a value to the endpoint URL.
{% /codeInfo %}
{% codeInfo srNumber=5 %}
**awsSessionToken**: The AWS session token is an optional parameter. If you want, enter the details of your temporary session token.
{% /codeInfo %}
{% partial file="/v1.7/connectors/yaml/common/aws-config-def.md" /%}
{% codeInfo srNumber=6 %}
@ -130,21 +102,9 @@ source:
type: Glue
awsConfig:
```
```yaml {% srNumber=1 %}
awsAccessKeyId: KEY
```
```yaml {% srNumber=2 %}
awsSecretAccessKey: SECRET
```
```yaml {% srNumber=3 %}
awsRegion: us-east-2
```
```yaml {% srNumber=4 %}
# endPointURL: https://glue.us-east-2.amazonaws.com/
```
```yaml {% srNumber=5 %}
# awsSessionToken: TOKEN
```
{% partial file="/v1.7/connectors/yaml/common/aws-config.md" /%}
```yaml {% srNumber=6 %}
databaseName: database_name
```

12
openmetadata-docs/content/v1.7.x-SNAPSHOT/connectors/database/s3-datalake/yaml.md
View File

@ -90,13 +90,11 @@ The workflow is modeled around the following JSON Schema.
{% codeInfo srNumber=1 %}
* **awsAccessKeyId**: Enter your secure access key ID for your DynamoDB connection. The specified key ID should be authorized to read all databases you want to include in the metadata ingestion workflow.
* **awsSecretAccessKey**: Enter the Secret Access Key (the passcode key pair to the key ID from above).
* **awsRegion**: Specify the region in which your DynamoDB is located. This setting is required even if you have configured a local AWS profile.
* **schemaFilterPattern** and **tableFilterPattern**: Note that the `schemaFilterPattern` and `tableFilterPattern` both support regex as `include` or `exclude`. E.g.,
{% /codeInfo %}
{% partial file="/v1.7/connectors/yaml/common/aws-config-def.md" /%}
{% partial file="/v1.7/connectors/yaml/database/source-config-def.md" /%}
@ -120,9 +118,11 @@ source:
```yaml {% srNumber=1 %}
configSource:
securityConfig:
awsAccessKeyId: aws access key id
awsSecretAccessKey: aws secret access key
awsRegion: aws region
```
{% partial file="/v1.7/connectors/yaml/common/aws-config.md" /%}
```yaml {% srNumber=1 %}
bucketName: bucket name
prefix: prefix
```

43
openmetadata-docs/content/v1.7.x-SNAPSHOT/connectors/ingestion/workflows/dbt/ingest-dbt-yaml.md
View File

@ -46,38 +46,7 @@ In this configuration we will be fetching the dbt `manifest.json`, `catalog.json
- **dbtConfigType**: s3
{% /codeInfo %}
{% codeInfo srNumber=1 %}
- **awsAccessKeyId** & **awsSecretAccessKey**: When you interact with AWS, you specify your AWS security credentials to verify who you are and whether you have
permission to access the resources that you are requesting. AWS uses the security credentials to authenticate and
authorize your requests ([docs](https://docs.aws.amazon.com/IAM/latest/UserGuide/security-creds.html)).
Access keys consist of two parts: An **access key ID** (for example, `AKIAIOSFODNN7EXAMPLE`), and a **secret access key** (for example, `wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY`).
You must use both the access key ID and secret access key together to authenticate your requests.
You can find further information on how to manage your access keys [here](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html).
{% /codeInfo %}
{% codeInfo srNumber=2 %}
**awsSessionToken**: If you are using temporary credentials to access your services, you will need to inform the AWS Access Key ID
and AWS Secrets Access Key. Also, these will include an AWS Session Token.
{% /codeInfo %}
{% codeInfo srNumber=3 %}
**awsRegion**: Each AWS Region is a separate geographic area in which AWS clusters data centers ([docs](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.RegionsAndAvailabilityZones.html)).
As AWS can have instances in multiple regions, we need to know the region the service you want reach belongs to.
Note that the AWS Region is the only required parameter when configuring a connection. When connecting to the
services programmatically, there are different ways in which we can extract and use the rest of AWS configurations.
You can find further information about configuring your credentials [here](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html#configuring-credentials).
{% /codeInfo %}
{% partial file="/v1.7/connectors/yaml/common/aws-config-def.md" /%}
{% partial file="/v1.7/connectors/yaml/dbt/dbt-prefix-def.md" /%}
@ -105,16 +74,10 @@ source:
```
```yaml {% srNumber=1 %}
dbtSecurityConfig:
awsAccessKeyId: AWS_ACCESS_KEY_ID
awsSecretAccessKey: AWS_SECRET_ACCESS_KEY
```
```yaml {% srNumber=2 %}
awsSessionToken: AWS_SESSION_TOKEN
```
```yaml {% srNumber=3 %}
awsRegion: AWS_REGION
```
{% partial file="/v1.7/connectors/yaml/common/aws-config.md" /%}
{% partial file="/v1.7/connectors/yaml/dbt/dbt-prefix.md" /%}
{% partial file="/v1.7/connectors/yaml/dbt/source-config.md" /%}

122
openmetadata-docs/content/v1.7.x-SNAPSHOT/connectors/messaging/kinesis/yaml.md
View File

@ -80,100 +80,7 @@ This is a sample config for Kinesis:
#### Source Configuration - Service Connection
{% codeInfo srNumber=1 %}
- **awsAccessKeyId** & **awsSecretAccessKey**: When you interact with AWS, you specify your AWS security credentials to verify who you are and whether you have
permission to access the resources that you are requesting. AWS uses the security credentials to authenticate and
authorize your requests ([docs](https://docs.aws.amazon.com/IAM/latest/UserGuide/security-creds.html)).
Access keys consist of two parts: An **access key ID** (for example, `AKIAIOSFODNN7EXAMPLE`), and a **secret access key** (for example, `wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY`).
You must use both the access key ID and secret access key together to authenticate your requests.
You can find further information on how to manage your access keys [here](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html).
{% /codeInfo %}
{% codeInfo srNumber=2 %}
**awsSessionToken**: If you are using temporary credentials to access your services, you will need to inform the AWS Access Key ID
and AWS Secrets Access Key. Also, these will include an AWS Session Token.
{% /codeInfo %}
{% codeInfo srNumber=3 %}
**awsRegion**: Each AWS Region is a separate geographic area in which AWS clusters data centers ([docs](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.RegionsAndAvailabilityZones.html)).
As AWS can have instances in multiple regions, we need to know the region the service you want reach belongs to.
Note that the AWS Region is the only required parameter when configuring a connection. When connecting to the
services programmatically, there are different ways in which we can extract and use the rest of AWS configurations.
You can find further information about configuring your credentials [here](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html#configuring-credentials).
{% /codeInfo %}
{% codeInfo srNumber=4 %}
**endPointURL**: To connect programmatically to an AWS service, you use an endpoint. An *endpoint* is the URL of the
entry point for an AWS web service. The AWS SDKs and the AWS Command Line Interface (AWS CLI) automatically use the
default endpoint for each service in an AWS Region. But you can specify an alternate endpoint for your API requests.
Find more information on [AWS service endpoints](https://docs.aws.amazon.com/general/latest/gr/rande.html).
{% /codeInfo %}
{% codeInfo srNumber=5 %}
**profileName**: A named profile is a collection of settings and credentials that you can apply to a AWS CLI command.
When you specify a profile to run a command, the settings and credentials are used to run that command.
Multiple named profiles can be stored in the config and credentials files.
You can inform this field if you'd like to use a profile other than `default`.
Find here more information about [Named profiles for the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-profiles.html).
{% /codeInfo %}
{% codeInfo srNumber=6 %}
**assumeRoleArn**: Typically, you use `AssumeRole` within your account or for cross-account access. In this field you'll set the
`ARN` (Amazon Resource Name) of the policy of the other account.
A user who wants to access a role in a different account must also have permissions that are delegated from the account
administrator. The administrator must attach a policy that allows the user to call `AssumeRole` for the `ARN` of the role in the other account.
This is a required field if you'd like to `AssumeRole`.
Find more information on [AssumeRole](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html).
{%note%}
When using Assume Role authentication, ensure you provide the following details:
- **AWS Region**: Specify the AWS region for your deployment.
- **Assume Role ARN**: Provide the ARN of the role in your AWS account that OpenMetadata will assume.
{%/note%}
{% /codeInfo %}
{% codeInfo srNumber=7 %}
**assumeRoleSessionName**: An identifier for the assumed role session. Use the role session name to uniquely identify a session when the same role
is assumed by different principals or for different reasons.
By default, we'll use the name `OpenMetadataSession`.
Find more information about the [Role Session Name](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html#:~:text=An%20identifier%20for%20the%20assumed%20role%20session.).
{% /codeInfo %}
{% codeInfo srNumber=8 %}
**assumeRoleSourceIdentity**: The source identity specified by the principal that is calling the `AssumeRole` operation. You can use source identity
information in AWS CloudTrail logs to determine who took actions with a role.
Find more information about [Source Identity](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html#:~:text=Required%3A%20No-,SourceIdentity,-The%20source%20identity).
{% /codeInfo %}
{% partial file="/v1.7/connectors/yaml/common/aws-config-def.md" /%}
{% partial file="/v1.7/connectors/yaml/messaging/source-config-def.md" /%}
@ -195,31 +102,8 @@ source:
type: Kinesis
awsConfig:
```
```yaml {% srNumber=1 %}
awsAccessKeyId: KEY
awsSecretAccessKey: SECRET
```
```yaml {% srNumber=2 %}
# awsSessionToken: TOKEN
```
```yaml {% srNumber=3 %}
awsRegion: us-east-2
```
```yaml {% srNumber=4 %}
# endPointURL: https://athena.us-east-2.amazonaws.com/custom
```
```yaml {% srNumber=5 %}
# profileName: profile
```
```yaml {% srNumber=6 %}
# assumeRoleArn: "arn:partition:service:region:account:resource"
```
```yaml {% srNumber=7 %}
# assumeRoleSessionName: session
```
```yaml {% srNumber=8 %}
# assumeRoleSourceIdentity: identity
```
{% partial file="/v1.7/connectors/yaml/common/aws-config.md" /%}
{% partial file="/v1.7/connectors/yaml/messaging/source-config.md" /%}

122
openmetadata-docs/content/v1.7.x-SNAPSHOT/connectors/ml-model/sagemaker/yaml.md
View File

@ -78,100 +78,7 @@ This is a sample config for Sagemaker:
#### Source Configuration - Service Connection
{% codeInfo srNumber=1 %}
- **awsAccessKeyId** & **awsSecretAccessKey**: When you interact with AWS, you specify your AWS security credentials to verify who you are and whether you have
permission to access the resources that you are requesting. AWS uses the security credentials to authenticate and
authorize your requests ([docs](https://docs.aws.amazon.com/IAM/latest/UserGuide/security-creds.html)).
Access keys consist of two parts: An **access key ID** (for example, `AKIAIOSFODNN7EXAMPLE`), and a **secret access key** (for example, `wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY`).
You must use both the access key ID and secret access key together to authenticate your requests.
You can find further information on how to manage your access keys [here](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html).
{% /codeInfo %}
{% codeInfo srNumber=2 %}
**awsSessionToken**: If you are using temporary credentials to access your services, you will need to inform the AWS Access Key ID
and AWS Secrets Access Key. Also, these will include an AWS Session Token.
{% /codeInfo %}
{% codeInfo srNumber=3 %}
**awsRegion**: Each AWS Region is a separate geographic area in which AWS clusters data centers ([docs](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.RegionsAndAvailabilityZones.html)).
As AWS can have instances in multiple regions, we need to know the region the service you want reach belongs to.
Note that the AWS Region is the only required parameter when configuring a connection. When connecting to the
services programmatically, there are different ways in which we can extract and use the rest of AWS configurations.
You can find further information about configuring your credentials [here](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html#configuring-credentials).
{% /codeInfo %}
{% codeInfo srNumber=4 %}
**endPointURL**: To connect programmatically to an AWS service, you use an endpoint. An *endpoint* is the URL of the
entry point for an AWS web service. The AWS SDKs and the AWS Command Line Interface (AWS CLI) automatically use the
default endpoint for each service in an AWS Region. But you can specify an alternate endpoint for your API requests.
Find more information on [AWS service endpoints](https://docs.aws.amazon.com/general/latest/gr/rande.html).
{% /codeInfo %}
{% codeInfo srNumber=5 %}
**profileName**: A named profile is a collection of settings and credentials that you can apply to a AWS CLI command.
When you specify a profile to run a command, the settings and credentials are used to run that command.
Multiple named profiles can be stored in the config and credentials files.
You can inform this field if you'd like to use a profile other than `default`.
Find here more information about [Named profiles for the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-profiles.html).
{% /codeInfo %}
{% codeInfo srNumber=6 %}
**assumeRoleArn**: Typically, you use `AssumeRole` within your account or for cross-account access. In this field you'll set the
`ARN` (Amazon Resource Name) of the policy of the other account.
A user who wants to access a role in a different account must also have permissions that are delegated from the account
administrator. The administrator must attach a policy that allows the user to call `AssumeRole` for the `ARN` of the role in the other account.
This is a required field if you'd like to `AssumeRole`.
Find more information on [AssumeRole](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html).
{%note%}
When using Assume Role authentication, ensure you provide the following details:
- **AWS Region**: Specify the AWS region for your deployment.
- **Assume Role ARN**: Provide the ARN of the role in your AWS account that OpenMetadata will assume.
{%/note%}
{% /codeInfo %}
{% codeInfo srNumber=7 %}
**assumeRoleSessionName**: An identifier for the assumed role session. Use the role session name to uniquely identify a session when the same role
is assumed by different principals or for different reasons.
By default, we'll use the name `OpenMetadataSession`.
Find more information about the [Role Session Name](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html#:~:text=An%20identifier%20for%20the%20assumed%20role%20session.).
{% /codeInfo %}
{% codeInfo srNumber=8 %}
**assumeRoleSourceIdentity**: The source identity specified by the principal that is calling the `AssumeRole` operation. You can use source identity
information in AWS CloudTrail logs to determine who took actions with a role.
Find more information about [Source Identity](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html#:~:text=Required%3A%20No-,SourceIdentity,-The%20source%20identity).
{% /codeInfo %}
{% partial file="/v1.7/connectors/yaml/common/aws-config-def.md" /%}
{% partial file="/v1.7/connectors/yaml/ml-model/source-config-def.md" /%}
@ -193,31 +100,8 @@ source:
type: SageMaker
awsConfig:
```
```yaml {% srNumber=1 %}
awsAccessKeyId: KEY
awsSecretAccessKey: SECRET
```
```yaml {% srNumber=2 %}
# awsSessionToken: TOKEN
```
```yaml {% srNumber=3 %}
awsRegion: us-east-2
```
```yaml {% srNumber=4 %}
# endPointURL: https://athena.us-east-2.amazonaws.com/custom
```
```yaml {% srNumber=5 %}
# profileName: profile
```
```yaml {% srNumber=6 %}
# assumeRoleArn: "arn:partition:service:region:account:resource"
```
```yaml {% srNumber=7 %}
# assumeRoleSessionName: session
```
```yaml {% srNumber=8 %}
# assumeRoleSourceIdentity: identity
```
{% partial file="/v1.7/connectors/yaml/common/aws-config.md" /%}
{% partial file="/v1.7/connectors/yaml/ml-model/source-config.md" /%}

82
openmetadata-docs/content/v1.7.x-SNAPSHOT/connectors/pipeline/glue-pipeline/yaml.md
View File

@ -63,58 +63,7 @@ This is a sample config for Glue:
#### Source Configuration - Service Connection
{% codeInfo srNumber=1 %}
**awsAccessKeyId**: Enter your secure access key ID for your Glue connection. The specified key ID should be
authorized to read all databases you want to include in the metadata ingestion workflow.
{% /codeInfo %}
{% codeInfo srNumber=2 %}
**awsSecretAccessKey**: Enter the Secret Access Key (the passcode key pair to the key ID from above).
{% /codeInfo %}
{% codeInfo srNumber=3 %}
**awsRegion**: Enter the location of the amazon cluster that your data and account are associated with.
{% /codeInfo %}
{% codeInfo srNumber=4 %}
**awsSessionToken**: The AWS session token is an optional parameter. If you want, enter the details of your temporary
session token.
{% /codeInfo %}
{% codeInfo srNumber=5 %}
**endPointURL**: Your Glue connector will automatically determine the AWS Glue endpoint URL based on the region. You
may override this behavior by entering a value to the endpoint URL.
{% /codeInfo %}
{% codeInfo srNumber=6 %}
**profileName**: The name of a profile to use with the boto session.
{% /codeInfo %}
{% codeInfo srNumber=7 %}
**assumeRoleArn**: The Amazon Resource Name (ARN) of the role to assume. Required Field in case of Assume Role.
{% /codeInfo %}
{% codeInfo srNumber=8 %}
**assumeRoleSessionName**: An identifier for the assumed role session. Use the role session name to uniquely identify a session when the same role is assumed by different principals or for different reasons. Required Field in case of Assume Role.
{% /codeInfo %}
{% codeInfo srNumber=9 %}
**assumeRoleSourceIdentity**: The Amazon Resource Name (ARN) of the role to assume. Optional Field in case of Assume Role.
{% /codeInfo %}
{% partial file="/v1.7/connectors/yaml/common/aws-config-def.md" /%}
{% partial file="/v1.6/connectors/yaml/pipeline/source-config-def.md" /%}
@ -135,34 +84,7 @@ source:
type: GluePipeline
awsConfig:
```
```yaml {% srNumber=1 %}
awsAccessKeyId: KEY
```
```yaml {% srNumber=2 %}
awsSecretAccessKey: SECRET
```
```yaml {% srNumber=3 %}
awsRegion: us-east-2
```
```yaml {% srNumber=4 %}
# awsSessionToken: TOKEN
```
```yaml {% srNumber=5 %}
# endPointURL: https://glue.us-east-2.amazonaws.com/
```
```yaml {% srNumber=6 %}
# profileName: ""
```
```yaml {% srNumber=7 %}
# assumeRoleArn: ""
```
```yaml {% srNumber=8 %}
# assumeRoleSessionName: OpenMetadataSession
```
```yaml {% srNumber=9 %}
# assumeRoleSourceIdentity : ""
```
{% partial file="/v1.7/connectors/yaml/common/aws-config.md" /%}
{% partial file="/v1.6/connectors/yaml/pipeline/source-config.md" /%}

123
openmetadata-docs/content/v1.7.x-SNAPSHOT/connectors/storage/s3/yaml.md
View File

@ -128,100 +128,7 @@ This is a sample config for Athena:
#### Source Configuration - Service Connection
{% codeInfo srNumber=1 %}
- **awsAccessKeyId** & **awsSecretAccessKey**: When you interact with AWS, you specify your AWS security credentials to verify who you are and whether you have
permission to access the resources that you are requesting. AWS uses the security credentials to authenticate and
authorize your requests ([docs](https://docs.aws.amazon.com/IAM/latest/UserGuide/security-creds.html)).
Access keys consist of two parts: An **access key ID** (for example, `AKIAIOSFODNN7EXAMPLE`), and a **secret access key** (for example, `wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY`).
You must use both the access key ID and secret access key together to authenticate your requests.
You can find further information on how to manage your access keys [here](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html).
{% /codeInfo %}
{% codeInfo srNumber=2 %}
**awsSessionToken**: If you are using temporary credentials to access your services, you will need to inform the AWS Access Key ID
and AWS Secrets Access Key. Also, these will include an AWS Session Token.
{% /codeInfo %}
{% codeInfo srNumber=3 %}
**awsRegion**: Each AWS Region is a separate geographic area in which AWS clusters data centers ([docs](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.RegionsAndAvailabilityZones.html)).
As AWS can have instances in multiple regions, we need to know the region the service you want reach belongs to.
Note that the AWS Region is the only required parameter when configuring a connection. When connecting to the
services programmatically, there are different ways in which we can extract and use the rest of AWS configurations.
You can find further information about configuring your credentials [here](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html#configuring-credentials).
{% /codeInfo %}
{% codeInfo srNumber=4 %}
**endPointURL**: To connect programmatically to an AWS service, you use an endpoint. An *endpoint* is the URL of the
entry point for an AWS web service. The AWS SDKs and the AWS Command Line Interface (AWS CLI) automatically use the
default endpoint for each service in an AWS Region. But you can specify an alternate endpoint for your API requests.
Find more information on [AWS service endpoints](https://docs.aws.amazon.com/general/latest/gr/rande.html).
{% /codeInfo %}
{% codeInfo srNumber=5 %}
**profileName**: A named profile is a collection of settings and credentials that you can apply to a AWS CLI command.
When you specify a profile to run a command, the settings and credentials are used to run that command.
Multiple named profiles can be stored in the config and credentials files.
You can inform this field if you'd like to use a profile other than `default`.
Find here more information about [Named profiles for the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-profiles.html).
{% /codeInfo %}
{% codeInfo srNumber=6 %}
**assumeRoleArn**: Typically, you use `AssumeRole` within your account or for cross-account access. In this field you'll set the
`ARN` (Amazon Resource Name) of the policy of the other account.
A user who wants to access a role in a different account must also have permissions that are delegated from the account
administrator. The administrator must attach a policy that allows the user to call `AssumeRole` for the `ARN` of the role in the other account.
This is a required field if you'd like to `AssumeRole`.
Find more information on [AssumeRole](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html).
{%note%}
When using Assume Role authentication, ensure you provide the following details:
- **AWS Region**: Specify the AWS region for your deployment.
- **Assume Role ARN**: Provide the ARN of the role in your AWS account that OpenMetadata will assume.
{%/note%}
{% /codeInfo %}
{% codeInfo srNumber=7 %}
**assumeRoleSessionName**: An identifier for the assumed role session. Use the role session name to uniquely identify a session when the same role
is assumed by different principals or for different reasons.
By default, we'll use the name `OpenMetadataSession`.
Find more information about the [Role Session Name](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html#:~:text=An%20identifier%20for%20the%20assumed%20role%20session.).
{% /codeInfo %}
{% codeInfo srNumber=8 %}
**assumeRoleSourceIdentity**: The source identity specified by the principal that is calling the `AssumeRole` operation. You can use source identity
information in AWS CloudTrail logs to determine who took actions with a role.
Find more information about [Source Identity](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html#:~:text=Required%3A%20No-,SourceIdentity,-The%20source%20identity).
{% /codeInfo %}
{% partial file="/v1.7/connectors/yaml/common/aws-config-def.md" /%}
{% partial file="/v1.7/connectors/yaml/storage/source-config-def.md" /%}
@ -262,31 +169,9 @@ source:
type: S3
awsConfig:
```
```yaml {% srNumber=1 %}
awsAccessKeyId: KEY
awsSecretAccessKey: SECRET
```
```yaml {% srNumber=2 %}
# awsSessionToken: TOKEN
```
```yaml {% srNumber=3 %}
awsRegion: us-east-2
```
```yaml {% srNumber=4 %}
# endPointURL: https://s3.us-east-2.amazonaws.com/custom
```
```yaml {% srNumber=5 %}
# profileName: profile
```
```yaml {% srNumber=6 %}
# assumeRoleArn: "arn:partition:service:region:account:resource"
```
```yaml {% srNumber=7 %}
# assumeRoleSessionName: session
```
```yaml {% srNumber=8 %}
# assumeRoleSourceIdentity: identity
```
{% partial file="/v1.7/connectors/yaml/common/aws-config.md" /%}
```yaml {% srNumber=13 %}
bucketNames:
- s3-testing-1