From d1ecf60acbacb90752b8336d425bb1c88a1f46e3 Mon Sep 17 00:00:00 2001 From: Rounak Dhillon <162090200+RounakDhillon@users.noreply.github.com> Date: Mon, 24 Mar 2025 09:07:16 +0530 Subject: [PATCH] Docs: Aws Partial Addition in Connectors (#20342) --- .../connectors/yaml/common/aws-config-def.md | 94 +++++++++++++ .../v1.6/connectors/yaml/common/aws-config.md | 25 ++++ .../connectors/yaml/common/aws-config-def.md | 94 +++++++++++++ .../v1.7/connectors/yaml/common/aws-config.md | 25 ++++ .../connectors/dashboard/quicksight/yaml.md | 21 +-- .../v1.6.x/connectors/database/athena/yaml.md | 122 +---------------- .../connectors/database/dynamodb/yaml.md | 48 +------ .../v1.6.x/connectors/database/glue/yaml.md | 48 +------ .../connectors/database/s3-datalake/yaml.md | 12 +- .../workflows/dbt/ingest-dbt-yaml.md | 43 +----- .../connectors/messaging/kinesis/yaml.md | 123 +----------------- .../connectors/ml-model/sagemaker/yaml.md | 122 +---------------- .../connectors/pipeline/glue-pipeline/yaml.md | 81 +----------- .../v1.6.x/connectors/storage/s3/yaml.md | 123 +----------------- .../connectors/dashboard/quicksight/yaml.md | 21 +-- .../connectors/database/athena/yaml.md | 123 +----------------- .../connectors/database/dynamodb/yaml.md | 48 +------ .../connectors/database/glue/yaml.md | 48 +------ .../connectors/database/s3-datalake/yaml.md | 12 +- .../workflows/dbt/ingest-dbt-yaml.md | 43 +----- .../connectors/messaging/kinesis/yaml.md | 122 +---------------- .../connectors/ml-model/sagemaker/yaml.md | 122 +---------------- .../connectors/pipeline/glue-pipeline/yaml.md | 82 +----------- .../connectors/storage/s3/yaml.md | 123 +----------------- 24 files changed, 313 insertions(+), 1412 deletions(-) create mode 100644 openmetadata-docs/content/partials/v1.6/connectors/yaml/common/aws-config-def.md create mode 100644 openmetadata-docs/content/partials/v1.6/connectors/yaml/common/aws-config.md create mode 100644 openmetadata-docs/content/partials/v1.7/connectors/yaml/common/aws-config-def.md create mode 100644 openmetadata-docs/content/partials/v1.7/connectors/yaml/common/aws-config.md diff --git a/openmetadata-docs/content/partials/v1.6/connectors/yaml/common/aws-config-def.md b/openmetadata-docs/content/partials/v1.6/connectors/yaml/common/aws-config-def.md new file mode 100644 index 00000000000..d3b643e6902 --- /dev/null +++ b/openmetadata-docs/content/partials/v1.6/connectors/yaml/common/aws-config-def.md @@ -0,0 +1,94 @@ +{% codeInfo srNumber=221 %} + +- **awsAccessKeyId** & **awsSecretAccessKey**: When you interact with AWS, you specify your AWS security credentials to verify who you are and whether you have + permission to access the resources that you are requesting. AWS uses the security credentials to authenticate and + authorize your requests ([docs](https://docs.aws.amazon.com/IAM/latest/UserGuide/security-creds.html)). + +Access keys consist of two parts: An **access key ID** (for example, `AKIAIOSFODNN7EXAMPLE`), and a **secret access key** (for example, `wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY`). + +You must use both the access key ID and secret access key together to authenticate your requests. + +You can find further information on how to manage your access keys [here](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html). + +{% /codeInfo %} + +{% codeInfo srNumber=222 %} +**awsSessionToken**: If you are using temporary credentials to access your services, you will need to inform the AWS Access Key ID +and AWS Secrets Access Key. Also, these will include an AWS Session Token. + +{% /codeInfo %} + +{% codeInfo srNumber=223 %} + +**awsRegion**: Each AWS Region is a separate geographic area in which AWS clusters data centers ([docs](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.RegionsAndAvailabilityZones.html)). + +As AWS can have instances in multiple regions, we need to know the region the service you want reach belongs to. + +Note that the AWS Region is the only required parameter when configuring a connection. When connecting to the +services programmatically, there are different ways in which we can extract and use the rest of AWS configurations. + +You can find further information about configuring your credentials [here](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html#configuring-credentials). + +{% /codeInfo %} + +{% codeInfo srNumber=224 %} + +**endPointURL**: To connect programmatically to an AWS service, you use an endpoint. An *endpoint* is the URL of the +entry point for an AWS web service. The AWS SDKs and the AWS Command Line Interface (AWS CLI) automatically use the +default endpoint for each service in an AWS Region. But you can specify an alternate endpoint for your API requests. + +Find more information on [AWS service endpoints](https://docs.aws.amazon.com/general/latest/gr/rande.html). + +{% /codeInfo %} + +{% codeInfo srNumber=225 %} + +**profileName**: A named profile is a collection of settings and credentials that you can apply to a AWS CLI command. +When you specify a profile to run a command, the settings and credentials are used to run that command. +Multiple named profiles can be stored in the config and credentials files. + +You can inform this field if you'd like to use a profile other than `default`. + +Find here more information about [Named profiles for the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-profiles.html). + +{% /codeInfo %} + +{% codeInfo srNumber=226 %} + +**assumeRoleArn**: Typically, you use `AssumeRole` within your account or for cross-account access. In this field you'll set the +`ARN` (Amazon Resource Name) of the policy of the other account. + +A user who wants to access a role in a different account must also have permissions that are delegated from the account +administrator. The administrator must attach a policy that allows the user to call `AssumeRole` for the `ARN` of the role in the other account. + +This is a required field if you'd like to `AssumeRole`. + +Find more information on [AssumeRole](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html). + +{%note%} +When using Assume Role authentication, ensure you provide the following details: +- **AWS Region**: Specify the AWS region for your deployment. +- **Assume Role ARN**: Provide the ARN of the role in your AWS account that OpenMetadata will assume. +{%/note%} + +{% /codeInfo %} + +{% codeInfo srNumber=227 %} + +**assumeRoleSessionName**: An identifier for the assumed role session. Use the role session name to uniquely identify a session when the same role +is assumed by different principals or for different reasons. + +By default, we'll use the name `OpenMetadataSession`. + +Find more information about the [Role Session Name](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html#:~:text=An%20identifier%20for%20the%20assumed%20role%20session.). + +{% /codeInfo %} + +{% codeInfo srNumber=228 %} + +**assumeRoleSourceIdentity**: The source identity specified by the principal that is calling the `AssumeRole` operation. You can use source identity +information in AWS CloudTrail logs to determine who took actions with a role. + +Find more information about [Source Identity](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html#:~:text=Required%3A%20No-,SourceIdentity,-The%20source%20identity). + +{% /codeInfo %} diff --git a/openmetadata-docs/content/partials/v1.6/connectors/yaml/common/aws-config.md b/openmetadata-docs/content/partials/v1.6/connectors/yaml/common/aws-config.md new file mode 100644 index 00000000000..bd5cef2eca1 --- /dev/null +++ b/openmetadata-docs/content/partials/v1.6/connectors/yaml/common/aws-config.md @@ -0,0 +1,25 @@ +```yaml {% srNumber=221 %} + awsAccessKeyId: KEY + awsSecretAccessKey: SECRET +``` +```yaml {% srNumber=222 %} + # awsSessionToken: TOKEN +``` +```yaml {% srNumber=223 %} + awsRegion: us-east-2 +``` +```yaml {% srNumber=224 %} + # endPointURL: https://athena.us-east-2.amazonaws.com/custom +``` +```yaml {% srNumber=225 %} + # profileName: profile +``` +```yaml {% srNumber=226 %} + # assumeRoleArn: "arn:partition:service:region:account:resource" +``` +```yaml {% srNumber=227 %} + # assumeRoleSessionName: session +``` +```yaml {% srNumber=228 %} + # assumeRoleSourceIdentity: identity +``` diff --git a/openmetadata-docs/content/partials/v1.7/connectors/yaml/common/aws-config-def.md b/openmetadata-docs/content/partials/v1.7/connectors/yaml/common/aws-config-def.md new file mode 100644 index 00000000000..d3b643e6902 --- /dev/null +++ b/openmetadata-docs/content/partials/v1.7/connectors/yaml/common/aws-config-def.md @@ -0,0 +1,94 @@ +{% codeInfo srNumber=221 %} + +- **awsAccessKeyId** & **awsSecretAccessKey**: When you interact with AWS, you specify your AWS security credentials to verify who you are and whether you have + permission to access the resources that you are requesting. AWS uses the security credentials to authenticate and + authorize your requests ([docs](https://docs.aws.amazon.com/IAM/latest/UserGuide/security-creds.html)). + +Access keys consist of two parts: An **access key ID** (for example, `AKIAIOSFODNN7EXAMPLE`), and a **secret access key** (for example, `wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY`). + +You must use both the access key ID and secret access key together to authenticate your requests. + +You can find further information on how to manage your access keys [here](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html). + +{% /codeInfo %} + +{% codeInfo srNumber=222 %} +**awsSessionToken**: If you are using temporary credentials to access your services, you will need to inform the AWS Access Key ID +and AWS Secrets Access Key. Also, these will include an AWS Session Token. + +{% /codeInfo %} + +{% codeInfo srNumber=223 %} + +**awsRegion**: Each AWS Region is a separate geographic area in which AWS clusters data centers ([docs](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.RegionsAndAvailabilityZones.html)). + +As AWS can have instances in multiple regions, we need to know the region the service you want reach belongs to. + +Note that the AWS Region is the only required parameter when configuring a connection. When connecting to the +services programmatically, there are different ways in which we can extract and use the rest of AWS configurations. + +You can find further information about configuring your credentials [here](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html#configuring-credentials). + +{% /codeInfo %} + +{% codeInfo srNumber=224 %} + +**endPointURL**: To connect programmatically to an AWS service, you use an endpoint. An *endpoint* is the URL of the +entry point for an AWS web service. The AWS SDKs and the AWS Command Line Interface (AWS CLI) automatically use the +default endpoint for each service in an AWS Region. But you can specify an alternate endpoint for your API requests. + +Find more information on [AWS service endpoints](https://docs.aws.amazon.com/general/latest/gr/rande.html). + +{% /codeInfo %} + +{% codeInfo srNumber=225 %} + +**profileName**: A named profile is a collection of settings and credentials that you can apply to a AWS CLI command. +When you specify a profile to run a command, the settings and credentials are used to run that command. +Multiple named profiles can be stored in the config and credentials files. + +You can inform this field if you'd like to use a profile other than `default`. + +Find here more information about [Named profiles for the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-profiles.html). + +{% /codeInfo %} + +{% codeInfo srNumber=226 %} + +**assumeRoleArn**: Typically, you use `AssumeRole` within your account or for cross-account access. In this field you'll set the +`ARN` (Amazon Resource Name) of the policy of the other account. + +A user who wants to access a role in a different account must also have permissions that are delegated from the account +administrator. The administrator must attach a policy that allows the user to call `AssumeRole` for the `ARN` of the role in the other account. + +This is a required field if you'd like to `AssumeRole`. + +Find more information on [AssumeRole](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html). + +{%note%} +When using Assume Role authentication, ensure you provide the following details: +- **AWS Region**: Specify the AWS region for your deployment. +- **Assume Role ARN**: Provide the ARN of the role in your AWS account that OpenMetadata will assume. +{%/note%} + +{% /codeInfo %} + +{% codeInfo srNumber=227 %} + +**assumeRoleSessionName**: An identifier for the assumed role session. Use the role session name to uniquely identify a session when the same role +is assumed by different principals or for different reasons. + +By default, we'll use the name `OpenMetadataSession`. + +Find more information about the [Role Session Name](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html#:~:text=An%20identifier%20for%20the%20assumed%20role%20session.). + +{% /codeInfo %} + +{% codeInfo srNumber=228 %} + +**assumeRoleSourceIdentity**: The source identity specified by the principal that is calling the `AssumeRole` operation. You can use source identity +information in AWS CloudTrail logs to determine who took actions with a role. + +Find more information about [Source Identity](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html#:~:text=Required%3A%20No-,SourceIdentity,-The%20source%20identity). + +{% /codeInfo %} diff --git a/openmetadata-docs/content/partials/v1.7/connectors/yaml/common/aws-config.md b/openmetadata-docs/content/partials/v1.7/connectors/yaml/common/aws-config.md new file mode 100644 index 00000000000..bd5cef2eca1 --- /dev/null +++ b/openmetadata-docs/content/partials/v1.7/connectors/yaml/common/aws-config.md @@ -0,0 +1,25 @@ +```yaml {% srNumber=221 %} + awsAccessKeyId: KEY + awsSecretAccessKey: SECRET +``` +```yaml {% srNumber=222 %} + # awsSessionToken: TOKEN +``` +```yaml {% srNumber=223 %} + awsRegion: us-east-2 +``` +```yaml {% srNumber=224 %} + # endPointURL: https://athena.us-east-2.amazonaws.com/custom +``` +```yaml {% srNumber=225 %} + # profileName: profile +``` +```yaml {% srNumber=226 %} + # assumeRoleArn: "arn:partition:service:region:account:resource" +``` +```yaml {% srNumber=227 %} + # assumeRoleSessionName: session +``` +```yaml {% srNumber=228 %} + # assumeRoleSourceIdentity: identity +``` diff --git a/openmetadata-docs/content/v1.6.x/connectors/dashboard/quicksight/yaml.md b/openmetadata-docs/content/v1.6.x/connectors/dashboard/quicksight/yaml.md index d85000e9826..d6bee5df54d 100644 --- a/openmetadata-docs/content/v1.6.x/connectors/dashboard/quicksight/yaml.md +++ b/openmetadata-docs/content/v1.6.x/connectors/dashboard/quicksight/yaml.md @@ -103,16 +103,7 @@ This is a sample config for QuickSight: #### Source Configuration - Service Connection -{% codeInfo srNumber=1 %} - -**awsConfig** - - **AWS Access Key ID**: Enter your secure access key ID for your Glue connection. The specified key ID should be authorized to read all databases you want to include in the metadata ingestion workflow. - - **AWS Secret Access Key**: Enter the Secret Access Key (the passcode key pair to the key ID from above). - - **AWS Region**: Enter the location of the amazon cluster that your data and account are associated with. - - **AWS Session Token (optional)**: The AWS session token is an optional parameter. If you want, enter the details of your temporary session token. - - **Endpoint URL (optional)**: Your Glue connector will automatically determine the AWS QuickSight endpoint URL based on the region. You may override this behavior by entering a value to the endpoint URL. - -{% /codeInfo %} +{% partial file="/v1.6/connectors/yaml/common/aws-config-def.md" /%} {% codeInfo srNumber=2 %} @@ -150,13 +141,9 @@ source: config: type: QuickSight ``` -```yaml {% srNumber=1 %} - awsConfig: - awsAccessKeyId: KEY - awsSecretAccessKey: SECRET - awsRegion: us-east-2 - awsSessionToken: Token -``` + +{% partial file="/v1.6/connectors/yaml/common/aws-config.md" /%} + ```yaml {% srNumber=2 %} awsAccountId: ``` diff --git a/openmetadata-docs/content/v1.6.x/connectors/database/athena/yaml.md b/openmetadata-docs/content/v1.6.x/connectors/database/athena/yaml.md index 4cbec75ab81..5c2107b793d 100644 --- a/openmetadata-docs/content/v1.6.x/connectors/database/athena/yaml.md +++ b/openmetadata-docs/content/v1.6.x/connectors/database/athena/yaml.md @@ -172,99 +172,7 @@ This is a sample config for Athena: #### Source Configuration - Service Connection -{% codeInfo srNumber=1 %} - -- **awsAccessKeyId** & **awsSecretAccessKey**: When you interact with AWS, you specify your AWS security credentials to verify who you are and whether you have - permission to access the resources that you are requesting. AWS uses the security credentials to authenticate and - authorize your requests ([docs](https://docs.aws.amazon.com/IAM/latest/UserGuide/security-creds.html)). - -Access keys consist of two parts: An **access key ID** (for example, `AKIAIOSFODNN7EXAMPLE`), and a **secret access key** (for example, `wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY`). - -You must use both the access key ID and secret access key together to authenticate your requests. - -You can find further information on how to manage your access keys [here](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html). - -{% /codeInfo %} - -{% codeInfo srNumber=2 %} -**awsSessionToken**: If you are using temporary credentials to access your services, you will need to inform the AWS Access Key ID -and AWS Secrets Access Key. Also, these will include an AWS Session Token. - -{% /codeInfo %} - -{% codeInfo srNumber=3 %} - -**awsRegion**: Each AWS Region is a separate geographic area in which AWS clusters data centers ([docs](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.RegionsAndAvailabilityZones.html)). - -As AWS can have instances in multiple regions, we need to know the region the service you want reach belongs to. - -Note that the AWS Region is the only required parameter when configuring a connection. When connecting to the -services programmatically, there are different ways in which we can extract and use the rest of AWS configurations. - -You can find further information about configuring your credentials [here](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html#configuring-credentials). - -{% /codeInfo %} - -{% codeInfo srNumber=4 %} - -**endPointURL**: To connect programmatically to an AWS service, you use an endpoint. An *endpoint* is the URL of the -entry point for an AWS web service. The AWS SDKs and the AWS Command Line Interface (AWS CLI) automatically use the -default endpoint for each service in an AWS Region. But you can specify an alternate endpoint for your API requests. - -Find more information on [AWS service endpoints](https://docs.aws.amazon.com/general/latest/gr/rande.html). - -{% /codeInfo %} - -{% codeInfo srNumber=5 %} - -**profileName**: A named profile is a collection of settings and credentials that you can apply to a AWS CLI command. -When you specify a profile to run a command, the settings and credentials are used to run that command. -Multiple named profiles can be stored in the config and credentials files. - -You can inform this field if you'd like to use a profile other than `default`. - -Find here more information about [Named profiles for the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-profiles.html). - -{% /codeInfo %} - -{% codeInfo srNumber=6 %} - -**assumeRoleArn**: Typically, you use `AssumeRole` within your account or for cross-account access. In this field you'll set the -`ARN` (Amazon Resource Name) of the policy of the other account. - -A user who wants to access a role in a different account must also have permissions that are delegated from the account -administrator. The administrator must attach a policy that allows the user to call `AssumeRole` for the `ARN` of the role in the other account. - -This is a required field if you'd like to `AssumeRole`. - -Find more information on [AssumeRole](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html). -{% /codeInfo %} - -{%note%} -When using Assume Role authentication, ensure you provide the following details: -- **AWS Region**: Specify the AWS region for your deployment. -- **Assume Role ARN**: Provide the ARN of the role in your AWS account that OpenMetadata will assume. -{%/note%} - -{% codeInfo srNumber=7 %} - -**assumeRoleSessionName**: An identifier for the assumed role session. Use the role session name to uniquely identify a session when the same role -is assumed by different principals or for different reasons. - -By default, we'll use the name `OpenMetadataSession`. - -Find more information about the [Role Session Name](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html#:~:text=An%20identifier%20for%20the%20assumed%20role%20session.). - -{% /codeInfo %} - -{% codeInfo srNumber=8 %} - -**assumeRoleSourceIdentity**: The source identity specified by the principal that is calling the `AssumeRole` operation. You can use source identity -information in AWS CloudTrail logs to determine who took actions with a role. - -Find more information about [Source Identity](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html#:~:text=Required%3A%20No-,SourceIdentity,-The%20source%20identity). - -{% /codeInfo %} +{% partial file="/v1.6/connectors/yaml/common/aws-config-def.md" /%} {% codeInfo srNumber=9 %} @@ -311,31 +219,9 @@ source: type: Athena awsConfig: ``` -```yaml {% srNumber=1 %} - awsAccessKeyId: KEY - awsSecretAccessKey: SECRET -``` -```yaml {% srNumber=2 %} - # awsSessionToken: TOKEN -``` -```yaml {% srNumber=3 %} - awsRegion: us-east-2 -``` -```yaml {% srNumber=4 %} - # endPointURL: https://athena.us-east-2.amazonaws.com/custom -``` -```yaml {% srNumber=5 %} - # profileName: profile -``` -```yaml {% srNumber=6 %} - # assumeRoleArn: "arn:partition:service:region:account:resource" -``` -```yaml {% srNumber=7 %} - # assumeRoleSessionName: session -``` -```yaml {% srNumber=8 %} - # assumeRoleSourceIdentity: identity -``` + +{% partial file="/v1.6/connectors/yaml/common/aws-config.md" /%} + ```yaml {% srNumber=9 %} s3StagingDir: s3 directory for datasource ``` diff --git a/openmetadata-docs/content/v1.6.x/connectors/database/dynamodb/yaml.md b/openmetadata-docs/content/v1.6.x/connectors/database/dynamodb/yaml.md index 9cfcaa3daa5..c3bec53f456 100644 --- a/openmetadata-docs/content/v1.6.x/connectors/database/dynamodb/yaml.md +++ b/openmetadata-docs/content/v1.6.x/connectors/database/dynamodb/yaml.md @@ -78,35 +78,7 @@ This is a sample config for DynamoDB: #### Source Configuration - Service Connection -{% codeInfo srNumber=1 %} - -**awsAccessKeyId**: Enter your secure access key ID for your DynamoDB connection. The specified key ID should be authorized to read all databases you want to include in the metadata ingestion workflow. - -{% /codeInfo %} - -{% codeInfo srNumber=2 %} - -**awsSecretAccessKey**: Enter the Secret Access Key (the passcode key pair to the key ID from above). - -{% /codeInfo %} - -{% codeInfo srNumber=3 %} - -**awsSessionToken**: The AWS session token is an optional parameter. If you want, enter the details of your temporary session token. - -{% /codeInfo %} - -{% codeInfo srNumber=4 %} - -**awsRegion**: Enter the location of the amazon cluster that your data and account are associated with. - -{% /codeInfo %} - -{% codeInfo srNumber=5 %} - -**endPointURL**: Your DynamoDB connector will automatically determine the AWS DynamoDB endpoint URL based on the region. You may override this behavior by entering a value to the endpoint URL. - -{% /codeInfo %} +{% partial file="/v1.6/connectors/yaml/common/aws-config-def.md" /%} {% codeInfo srNumber=6 %} @@ -149,21 +121,9 @@ source: type: DynamoDB awsConfig: ``` -```yaml {% srNumber=1 %} - awsAccessKeyId: aws_access_key_id -``` -```yaml {% srNumber=2 %} - awsSecretAccessKey: aws_secret_access_key -``` -```yaml {% srNumber=3 %} - awsSessionToken: AWS Session Token -``` -```yaml {% srNumber=4 %} - awsRegion: aws region -``` -```yaml {% srNumber=5 %} - endPointURL: https://dynamodb..amazonaws.com -``` + +{% partial file="/v1.6/connectors/yaml/common/aws-config.md" /%} + ```yaml {% srNumber=6 %} database: custom_database_name ``` diff --git a/openmetadata-docs/content/v1.6.x/connectors/database/glue/yaml.md b/openmetadata-docs/content/v1.6.x/connectors/database/glue/yaml.md index 7240059aca1..7fb09db6039 100644 --- a/openmetadata-docs/content/v1.6.x/connectors/database/glue/yaml.md +++ b/openmetadata-docs/content/v1.6.x/connectors/database/glue/yaml.md @@ -58,35 +58,7 @@ This is a sample config for Glue: #### Source Configuration - Service Connection -{% codeInfo srNumber=1 %} - -**awsAccessKeyId**: Enter your secure access key ID for your Glue connection. The specified key ID should be authorized to read all databases you want to include in the metadata ingestion workflow. - -{% /codeInfo %} - -{% codeInfo srNumber=2 %} - -**awsSecretAccessKey**: Enter the Secret Access Key (the passcode key pair to the key ID from above). - -{% /codeInfo %} - -{% codeInfo srNumber=3 %} - -**awsRegion**: Enter the location of the amazon cluster that your data and account are associated with. - -{% /codeInfo %} - -{% codeInfo srNumber=4 %} - -**endPointURL**: Your Glue connector will automatically determine the AWS Glue endpoint URL based on the region. You may override this behavior by entering a value to the endpoint URL. - -{% /codeInfo %} - -{% codeInfo srNumber=5 %} - -**awsSessionToken**: The AWS session token is an optional parameter. If you want, enter the details of your temporary session token. - -{% /codeInfo %} +{% partial file="/v1.6/connectors/yaml/common/aws-config-def.md" /%} {% codeInfo srNumber=6 %} @@ -130,21 +102,9 @@ source: type: Glue awsConfig: ``` -```yaml {% srNumber=1 %} - awsAccessKeyId: KEY -``` -```yaml {% srNumber=2 %} - awsSecretAccessKey: SECRET -``` -```yaml {% srNumber=3 %} - awsRegion: us-east-2 -``` -```yaml {% srNumber=4 %} - # endPointURL: https://glue.us-east-2.amazonaws.com/ -``` -```yaml {% srNumber=5 %} - # awsSessionToken: TOKEN -``` + +{% partial file="/v1.6/connectors/yaml/common/aws-config.md" /%} + ```yaml {% srNumber=6 %} databaseName: database_name ``` diff --git a/openmetadata-docs/content/v1.6.x/connectors/database/s3-datalake/yaml.md b/openmetadata-docs/content/v1.6.x/connectors/database/s3-datalake/yaml.md index 37fa714f983..9bde7a1341c 100644 --- a/openmetadata-docs/content/v1.6.x/connectors/database/s3-datalake/yaml.md +++ b/openmetadata-docs/content/v1.6.x/connectors/database/s3-datalake/yaml.md @@ -90,13 +90,11 @@ The workflow is modeled around the following JSON Schema. {% codeInfo srNumber=1 %} -* **awsAccessKeyId**: Enter your secure access key ID for your DynamoDB connection. The specified key ID should be authorized to read all databases you want to include in the metadata ingestion workflow. -* **awsSecretAccessKey**: Enter the Secret Access Key (the passcode key pair to the key ID from above). -* **awsRegion**: Specify the region in which your DynamoDB is located. This setting is required even if you have configured a local AWS profile. * **schemaFilterPattern** and **tableFilterPattern**: Note that the `schemaFilterPattern` and `tableFilterPattern` both support regex as `include` or `exclude`. E.g., {% /codeInfo %} +{% partial file="/v1.6/connectors/yaml/common/aws-config-def.md" /%} {% partial file="/v1.6/connectors/yaml/database/source-config-def.md" /%} @@ -120,9 +118,11 @@ source: ```yaml {% srNumber=1 %} configSource: securityConfig: - awsAccessKeyId: aws access key id - awsSecretAccessKey: aws secret access key - awsRegion: aws region +``` + +{% partial file="/v1.6/connectors/yaml/common/aws-config.md" /%} + +```yaml {% srNumber=1 %} bucketName: bucket name prefix: prefix ``` diff --git a/openmetadata-docs/content/v1.6.x/connectors/ingestion/workflows/dbt/ingest-dbt-yaml.md b/openmetadata-docs/content/v1.6.x/connectors/ingestion/workflows/dbt/ingest-dbt-yaml.md index 3ae6c622bfe..9e111b81b89 100644 --- a/openmetadata-docs/content/v1.6.x/connectors/ingestion/workflows/dbt/ingest-dbt-yaml.md +++ b/openmetadata-docs/content/v1.6.x/connectors/ingestion/workflows/dbt/ingest-dbt-yaml.md @@ -46,38 +46,7 @@ In this configuration we will be fetching the dbt `manifest.json`, `catalog.json - **dbtConfigType**: s3 {% /codeInfo %} -{% codeInfo srNumber=1 %} - -- **awsAccessKeyId** & **awsSecretAccessKey**: When you interact with AWS, you specify your AWS security credentials to verify who you are and whether you have - permission to access the resources that you are requesting. AWS uses the security credentials to authenticate and - authorize your requests ([docs](https://docs.aws.amazon.com/IAM/latest/UserGuide/security-creds.html)). - -Access keys consist of two parts: An **access key ID** (for example, `AKIAIOSFODNN7EXAMPLE`), and a **secret access key** (for example, `wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY`). - -You must use both the access key ID and secret access key together to authenticate your requests. - -You can find further information on how to manage your access keys [here](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html). - -{% /codeInfo %} - -{% codeInfo srNumber=2 %} -**awsSessionToken**: If you are using temporary credentials to access your services, you will need to inform the AWS Access Key ID -and AWS Secrets Access Key. Also, these will include an AWS Session Token. - -{% /codeInfo %} - -{% codeInfo srNumber=3 %} - -**awsRegion**: Each AWS Region is a separate geographic area in which AWS clusters data centers ([docs](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.RegionsAndAvailabilityZones.html)). - -As AWS can have instances in multiple regions, we need to know the region the service you want reach belongs to. - -Note that the AWS Region is the only required parameter when configuring a connection. When connecting to the -services programmatically, there are different ways in which we can extract and use the rest of AWS configurations. - -You can find further information about configuring your credentials [here](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html#configuring-credentials). - -{% /codeInfo %} +{% partial file="/v1.6/connectors/yaml/common/aws-config-def.md" /%} {% partial file="/v1.6/connectors/yaml/dbt/dbt-prefix-def.md" /%} @@ -105,16 +74,10 @@ source: ``` ```yaml {% srNumber=1 %} dbtSecurityConfig: - awsAccessKeyId: AWS_ACCESS_KEY_ID - awsSecretAccessKey: AWS_SECRET_ACCESS_KEY -``` -```yaml {% srNumber=2 %} - awsSessionToken: AWS_SESSION_TOKEN -``` -```yaml {% srNumber=3 %} - awsRegion: AWS_REGION ``` +{% partial file="/v1.6/connectors/yaml/common/aws-config.md" /%} + {% partial file="/v1.6/connectors/yaml/dbt/dbt-prefix.md" /%} {% partial file="/v1.6/connectors/yaml/dbt/source-config.md" /%} diff --git a/openmetadata-docs/content/v1.6.x/connectors/messaging/kinesis/yaml.md b/openmetadata-docs/content/v1.6.x/connectors/messaging/kinesis/yaml.md index fee14d97a48..8e5052bf7ad 100644 --- a/openmetadata-docs/content/v1.6.x/connectors/messaging/kinesis/yaml.md +++ b/openmetadata-docs/content/v1.6.x/connectors/messaging/kinesis/yaml.md @@ -80,100 +80,7 @@ This is a sample config for Kinesis: #### Source Configuration - Service Connection -{% codeInfo srNumber=1 %} - -- **awsAccessKeyId** & **awsSecretAccessKey**: When you interact with AWS, you specify your AWS security credentials to verify who you are and whether you have - permission to access the resources that you are requesting. AWS uses the security credentials to authenticate and - authorize your requests ([docs](https://docs.aws.amazon.com/IAM/latest/UserGuide/security-creds.html)). - -Access keys consist of two parts: An **access key ID** (for example, `AKIAIOSFODNN7EXAMPLE`), and a **secret access key** (for example, `wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY`). - -You must use both the access key ID and secret access key together to authenticate your requests. - -You can find further information on how to manage your access keys [here](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html). - -{% /codeInfo %} - -{% codeInfo srNumber=2 %} -**awsSessionToken**: If you are using temporary credentials to access your services, you will need to inform the AWS Access Key ID -and AWS Secrets Access Key. Also, these will include an AWS Session Token. - -{% /codeInfo %} - -{% codeInfo srNumber=3 %} - -**awsRegion**: Each AWS Region is a separate geographic area in which AWS clusters data centers ([docs](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.RegionsAndAvailabilityZones.html)). - -As AWS can have instances in multiple regions, we need to know the region the service you want reach belongs to. - -Note that the AWS Region is the only required parameter when configuring a connection. When connecting to the -services programmatically, there are different ways in which we can extract and use the rest of AWS configurations. - -You can find further information about configuring your credentials [here](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html#configuring-credentials). - -{% /codeInfo %} - -{% codeInfo srNumber=4 %} - -**endPointURL**: To connect programmatically to an AWS service, you use an endpoint. An *endpoint* is the URL of the -entry point for an AWS web service. The AWS SDKs and the AWS Command Line Interface (AWS CLI) automatically use the -default endpoint for each service in an AWS Region. But you can specify an alternate endpoint for your API requests. - -Find more information on [AWS service endpoints](https://docs.aws.amazon.com/general/latest/gr/rande.html). - -{% /codeInfo %} - -{% codeInfo srNumber=5 %} - -**profileName**: A named profile is a collection of settings and credentials that you can apply to a AWS CLI command. -When you specify a profile to run a command, the settings and credentials are used to run that command. -Multiple named profiles can be stored in the config and credentials files. - -You can inform this field if you'd like to use a profile other than `default`. - -Find here more information about [Named profiles for the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-profiles.html). - -{% /codeInfo %} - -{% codeInfo srNumber=6 %} - -**assumeRoleArn**: Typically, you use `AssumeRole` within your account or for cross-account access. In this field you'll set the -`ARN` (Amazon Resource Name) of the policy of the other account. - -A user who wants to access a role in a different account must also have permissions that are delegated from the account -administrator. The administrator must attach a policy that allows the user to call `AssumeRole` for the `ARN` of the role in the other account. - -This is a required field if you'd like to `AssumeRole`. - -Find more information on [AssumeRole](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html). - -{%note%} -When using Assume Role authentication, ensure you provide the following details: -- **AWS Region**: Specify the AWS region for your deployment. -- **Assume Role ARN**: Provide the ARN of the role in your AWS account that OpenMetadata will assume. -{%/note%} - -{% /codeInfo %} - -{% codeInfo srNumber=7 %} - -**assumeRoleSessionName**: An identifier for the assumed role session. Use the role session name to uniquely identify a session when the same role -is assumed by different principals or for different reasons. - -By default, we'll use the name `OpenMetadataSession`. - -Find more information about the [Role Session Name](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html#:~:text=An%20identifier%20for%20the%20assumed%20role%20session.). - -{% /codeInfo %} - -{% codeInfo srNumber=8 %} - -**assumeRoleSourceIdentity**: The source identity specified by the principal that is calling the `AssumeRole` operation. You can use source identity -information in AWS CloudTrail logs to determine who took actions with a role. - -Find more information about [Source Identity](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html#:~:text=Required%3A%20No-,SourceIdentity,-The%20source%20identity). - -{% /codeInfo %} +{% partial file="/v1.6/connectors/yaml/common/aws-config-def.md" /%} {% partial file="/v1.6/connectors/yaml/messaging/source-config-def.md" /%} @@ -195,31 +102,9 @@ source: type: Kinesis awsConfig: ``` -```yaml {% srNumber=1 %} - awsAccessKeyId: KEY - awsSecretAccessKey: SECRET -``` -```yaml {% srNumber=2 %} - # awsSessionToken: TOKEN -``` -```yaml {% srNumber=3 %} - awsRegion: us-east-2 -``` -```yaml {% srNumber=4 %} - # endPointURL: https://athena.us-east-2.amazonaws.com/custom -``` -```yaml {% srNumber=5 %} - # profileName: profile -``` -```yaml {% srNumber=6 %} - # assumeRoleArn: "arn:partition:service:region:account:resource" -``` -```yaml {% srNumber=7 %} - # assumeRoleSessionName: session -``` -```yaml {% srNumber=8 %} - # assumeRoleSourceIdentity: identity -``` + +{% partial file="/v1.6/connectors/yaml/common/aws-config.md" /%} + {% partial file="/v1.6/connectors/yaml/messaging/source-config.md" /%} diff --git a/openmetadata-docs/content/v1.6.x/connectors/ml-model/sagemaker/yaml.md b/openmetadata-docs/content/v1.6.x/connectors/ml-model/sagemaker/yaml.md index c7df7340ff7..ecba36f0a70 100644 --- a/openmetadata-docs/content/v1.6.x/connectors/ml-model/sagemaker/yaml.md +++ b/openmetadata-docs/content/v1.6.x/connectors/ml-model/sagemaker/yaml.md @@ -78,100 +78,7 @@ This is a sample config for Sagemaker: #### Source Configuration - Service Connection -{% codeInfo srNumber=1 %} - -- **awsAccessKeyId** & **awsSecretAccessKey**: When you interact with AWS, you specify your AWS security credentials to verify who you are and whether you have - permission to access the resources that you are requesting. AWS uses the security credentials to authenticate and - authorize your requests ([docs](https://docs.aws.amazon.com/IAM/latest/UserGuide/security-creds.html)). - -Access keys consist of two parts: An **access key ID** (for example, `AKIAIOSFODNN7EXAMPLE`), and a **secret access key** (for example, `wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY`). - -You must use both the access key ID and secret access key together to authenticate your requests. - -You can find further information on how to manage your access keys [here](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html). - -{% /codeInfo %} - -{% codeInfo srNumber=2 %} -**awsSessionToken**: If you are using temporary credentials to access your services, you will need to inform the AWS Access Key ID -and AWS Secrets Access Key. Also, these will include an AWS Session Token. - -{% /codeInfo %} - -{% codeInfo srNumber=3 %} - -**awsRegion**: Each AWS Region is a separate geographic area in which AWS clusters data centers ([docs](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.RegionsAndAvailabilityZones.html)). - -As AWS can have instances in multiple regions, we need to know the region the service you want reach belongs to. - -Note that the AWS Region is the only required parameter when configuring a connection. When connecting to the -services programmatically, there are different ways in which we can extract and use the rest of AWS configurations. - -You can find further information about configuring your credentials [here](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html#configuring-credentials). - -{% /codeInfo %} - -{% codeInfo srNumber=4 %} - -**endPointURL**: To connect programmatically to an AWS service, you use an endpoint. An *endpoint* is the URL of the -entry point for an AWS web service. The AWS SDKs and the AWS Command Line Interface (AWS CLI) automatically use the -default endpoint for each service in an AWS Region. But you can specify an alternate endpoint for your API requests. - -Find more information on [AWS service endpoints](https://docs.aws.amazon.com/general/latest/gr/rande.html). - -{% /codeInfo %} - -{% codeInfo srNumber=5 %} - -**profileName**: A named profile is a collection of settings and credentials that you can apply to a AWS CLI command. -When you specify a profile to run a command, the settings and credentials are used to run that command. -Multiple named profiles can be stored in the config and credentials files. - -You can inform this field if you'd like to use a profile other than `default`. - -Find here more information about [Named profiles for the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-profiles.html). - -{% /codeInfo %} - -{% codeInfo srNumber=6 %} - -**assumeRoleArn**: Typically, you use `AssumeRole` within your account or for cross-account access. In this field you'll set the -`ARN` (Amazon Resource Name) of the policy of the other account. - -A user who wants to access a role in a different account must also have permissions that are delegated from the account -administrator. The administrator must attach a policy that allows the user to call `AssumeRole` for the `ARN` of the role in the other account. - -This is a required field if you'd like to `AssumeRole`. - -Find more information on [AssumeRole](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html). - -{%note%} -When using Assume Role authentication, ensure you provide the following details: -- **AWS Region**: Specify the AWS region for your deployment. -- **Assume Role ARN**: Provide the ARN of the role in your AWS account that OpenMetadata will assume. -{%/note%} - -{% /codeInfo %} - -{% codeInfo srNumber=7 %} - -**assumeRoleSessionName**: An identifier for the assumed role session. Use the role session name to uniquely identify a session when the same role -is assumed by different principals or for different reasons. - -By default, we'll use the name `OpenMetadataSession`. - -Find more information about the [Role Session Name](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html#:~:text=An%20identifier%20for%20the%20assumed%20role%20session.). - -{% /codeInfo %} - -{% codeInfo srNumber=8 %} - -**assumeRoleSourceIdentity**: The source identity specified by the principal that is calling the `AssumeRole` operation. You can use source identity -information in AWS CloudTrail logs to determine who took actions with a role. - -Find more information about [Source Identity](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html#:~:text=Required%3A%20No-,SourceIdentity,-The%20source%20identity). - -{% /codeInfo %} +{% partial file="/v1.6/connectors/yaml/common/aws-config-def.md" /%} {% partial file="/v1.6/connectors/yaml/ml-model/source-config-def.md" /%} @@ -193,31 +100,8 @@ source: type: SageMaker awsConfig: ``` -```yaml {% srNumber=1 %} - awsAccessKeyId: KEY - awsSecretAccessKey: SECRET -``` -```yaml {% srNumber=2 %} - # awsSessionToken: TOKEN -``` -```yaml {% srNumber=3 %} - awsRegion: us-east-2 -``` -```yaml {% srNumber=4 %} - # endPointURL: https://athena.us-east-2.amazonaws.com/custom -``` -```yaml {% srNumber=5 %} - # profileName: profile -``` -```yaml {% srNumber=6 %} - # assumeRoleArn: "arn:partition:service:region:account:resource" -``` -```yaml {% srNumber=7 %} - # assumeRoleSessionName: session -``` -```yaml {% srNumber=8 %} - # assumeRoleSourceIdentity: identity -``` + +{% partial file="/v1.6/connectors/yaml/common/aws-config.md" /%} {% partial file="/v1.6/connectors/yaml/ml-model/source-config.md" /%} diff --git a/openmetadata-docs/content/v1.6.x/connectors/pipeline/glue-pipeline/yaml.md b/openmetadata-docs/content/v1.6.x/connectors/pipeline/glue-pipeline/yaml.md index 5ec3cc52cab..b239fc8a709 100644 --- a/openmetadata-docs/content/v1.6.x/connectors/pipeline/glue-pipeline/yaml.md +++ b/openmetadata-docs/content/v1.6.x/connectors/pipeline/glue-pipeline/yaml.md @@ -63,58 +63,7 @@ This is a sample config for Glue: #### Source Configuration - Service Connection -{% codeInfo srNumber=1 %} - - -**awsAccessKeyId**: Enter your secure access key ID for your Glue connection. The specified key ID should be - authorized to read all databases you want to include in the metadata ingestion workflow. - -{% /codeInfo %} - -{% codeInfo srNumber=2 %} - -**awsSecretAccessKey**: Enter the Secret Access Key (the passcode key pair to the key ID from above). - - -{% /codeInfo %} - -{% codeInfo srNumber=3 %} - -**awsRegion**: Enter the location of the amazon cluster that your data and account are associated with. - -{% /codeInfo %} - -{% codeInfo srNumber=4 %} - -**awsSessionToken**: The AWS session token is an optional parameter. If you want, enter the details of your temporary - session token. - -{% /codeInfo %} - -{% codeInfo srNumber=5 %} - -**endPointURL**: Your Glue connector will automatically determine the AWS Glue endpoint URL based on the region. You - may override this behavior by entering a value to the endpoint URL. - - -{% /codeInfo %} - -{% codeInfo srNumber=6 %} -**profileName**: The name of a profile to use with the boto session. -{% /codeInfo %} - -{% codeInfo srNumber=7 %} -**assumeRoleArn**: The Amazon Resource Name (ARN) of the role to assume. Required Field in case of Assume Role. -{% /codeInfo %} - -{% codeInfo srNumber=8 %} -**assumeRoleSessionName**: An identifier for the assumed role session. Use the role session name to uniquely identify a session when the same role is assumed by different principals or for different reasons. Required Field in case of Assume Role. -{% /codeInfo %} - -{% codeInfo srNumber=9 %} -**assumeRoleSourceIdentity**: The Amazon Resource Name (ARN) of the role to assume. Optional Field in case of Assume Role. -{% /codeInfo %} - +{% partial file="/v1.6/connectors/yaml/common/aws-config-def.md" /%} {% partial file="/v1.6/connectors/yaml/pipeline/source-config-def.md" /%} @@ -135,34 +84,8 @@ source: type: GluePipeline awsConfig: ``` -```yaml {% srNumber=1 %} - awsAccessKeyId: KEY -``` -```yaml {% srNumber=2 %} - awsSecretAccessKey: SECRET -``` -```yaml {% srNumber=3 %} - awsRegion: us-east-2 -``` -```yaml {% srNumber=4 %} - # awsSessionToken: TOKEN -``` -```yaml {% srNumber=5 %} - # endPointURL: https://glue.us-east-2.amazonaws.com/ -``` -```yaml {% srNumber=6 %} - # profileName: "" -``` -```yaml {% srNumber=7 %} - # assumeRoleArn: "" -``` -```yaml {% srNumber=8 %} - # assumeRoleSessionName: OpenMetadataSession -``` -```yaml {% srNumber=9 %} - # assumeRoleSourceIdentity : "" -``` +{% partial file="/v1.6/connectors/yaml/common/aws-config.md" /%} {% partial file="/v1.6/connectors/yaml/pipeline/source-config.md" /%} diff --git a/openmetadata-docs/content/v1.6.x/connectors/storage/s3/yaml.md b/openmetadata-docs/content/v1.6.x/connectors/storage/s3/yaml.md index 38ce78625a3..13000f58c9a 100644 --- a/openmetadata-docs/content/v1.6.x/connectors/storage/s3/yaml.md +++ b/openmetadata-docs/content/v1.6.x/connectors/storage/s3/yaml.md @@ -128,100 +128,7 @@ This is a sample config for Athena: #### Source Configuration - Service Connection -{% codeInfo srNumber=1 %} - -- **awsAccessKeyId** & **awsSecretAccessKey**: When you interact with AWS, you specify your AWS security credentials to verify who you are and whether you have - permission to access the resources that you are requesting. AWS uses the security credentials to authenticate and - authorize your requests ([docs](https://docs.aws.amazon.com/IAM/latest/UserGuide/security-creds.html)). - -Access keys consist of two parts: An **access key ID** (for example, `AKIAIOSFODNN7EXAMPLE`), and a **secret access key** (for example, `wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY`). - -You must use both the access key ID and secret access key together to authenticate your requests. - -You can find further information on how to manage your access keys [here](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html). - -{% /codeInfo %} - -{% codeInfo srNumber=2 %} -**awsSessionToken**: If you are using temporary credentials to access your services, you will need to inform the AWS Access Key ID -and AWS Secrets Access Key. Also, these will include an AWS Session Token. - -{% /codeInfo %} - -{% codeInfo srNumber=3 %} - -**awsRegion**: Each AWS Region is a separate geographic area in which AWS clusters data centers ([docs](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.RegionsAndAvailabilityZones.html)). - -As AWS can have instances in multiple regions, we need to know the region the service you want reach belongs to. - -Note that the AWS Region is the only required parameter when configuring a connection. When connecting to the -services programmatically, there are different ways in which we can extract and use the rest of AWS configurations. - -You can find further information about configuring your credentials [here](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html#configuring-credentials). - -{% /codeInfo %} - -{% codeInfo srNumber=4 %} - -**endPointURL**: To connect programmatically to an AWS service, you use an endpoint. An *endpoint* is the URL of the -entry point for an AWS web service. The AWS SDKs and the AWS Command Line Interface (AWS CLI) automatically use the -default endpoint for each service in an AWS Region. But you can specify an alternate endpoint for your API requests. - -Find more information on [AWS service endpoints](https://docs.aws.amazon.com/general/latest/gr/rande.html). - -{% /codeInfo %} - -{% codeInfo srNumber=5 %} - -**profileName**: A named profile is a collection of settings and credentials that you can apply to a AWS CLI command. -When you specify a profile to run a command, the settings and credentials are used to run that command. -Multiple named profiles can be stored in the config and credentials files. - -You can inform this field if you'd like to use a profile other than `default`. - -Find here more information about [Named profiles for the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-profiles.html). - -{% /codeInfo %} - -{% codeInfo srNumber=6 %} - -**assumeRoleArn**: Typically, you use `AssumeRole` within your account or for cross-account access. In this field you'll set the -`ARN` (Amazon Resource Name) of the policy of the other account. - -A user who wants to access a role in a different account must also have permissions that are delegated from the account -administrator. The administrator must attach a policy that allows the user to call `AssumeRole` for the `ARN` of the role in the other account. - -This is a required field if you'd like to `AssumeRole`. - -Find more information on [AssumeRole](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html). - -{%note%} -When using Assume Role authentication, ensure you provide the following details: -- **AWS Region**: Specify the AWS region for your deployment. -- **Assume Role ARN**: Provide the ARN of the role in your AWS account that OpenMetadata will assume. -{%/note%} - -{% /codeInfo %} - -{% codeInfo srNumber=7 %} - -**assumeRoleSessionName**: An identifier for the assumed role session. Use the role session name to uniquely identify a session when the same role -is assumed by different principals or for different reasons. - -By default, we'll use the name `OpenMetadataSession`. - -Find more information about the [Role Session Name](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html#:~:text=An%20identifier%20for%20the%20assumed%20role%20session.). - -{% /codeInfo %} - -{% codeInfo srNumber=8 %} - -**assumeRoleSourceIdentity**: The source identity specified by the principal that is calling the `AssumeRole` operation. You can use source identity -information in AWS CloudTrail logs to determine who took actions with a role. - -Find more information about [Source Identity](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html#:~:text=Required%3A%20No-,SourceIdentity,-The%20source%20identity). - -{% /codeInfo %} +{% partial file="/v1.6/connectors/yaml/common/aws-config-def.md" /%} {% partial file="/v1.6/connectors/yaml/storage/source-config-def.md" /%} @@ -262,31 +169,9 @@ source: type: S3 awsConfig: ``` -```yaml {% srNumber=1 %} - awsAccessKeyId: KEY - awsSecretAccessKey: SECRET -``` -```yaml {% srNumber=2 %} - # awsSessionToken: TOKEN -``` -```yaml {% srNumber=3 %} - awsRegion: us-east-2 -``` -```yaml {% srNumber=4 %} - # endPointURL: https://s3.us-east-2.amazonaws.com/custom -``` -```yaml {% srNumber=5 %} - # profileName: profile -``` -```yaml {% srNumber=6 %} - # assumeRoleArn: "arn:partition:service:region:account:resource" -``` -```yaml {% srNumber=7 %} - # assumeRoleSessionName: session -``` -```yaml {% srNumber=8 %} - # assumeRoleSourceIdentity: identity -``` + +{% partial file="/v1.6/connectors/yaml/common/aws-config.md" /%} + ```yaml {% srNumber=13 %} bucketNames: - s3-testing-1 diff --git a/openmetadata-docs/content/v1.7.x-SNAPSHOT/connectors/dashboard/quicksight/yaml.md b/openmetadata-docs/content/v1.7.x-SNAPSHOT/connectors/dashboard/quicksight/yaml.md index a3c993f45c8..e8804cc3ebd 100644 --- a/openmetadata-docs/content/v1.7.x-SNAPSHOT/connectors/dashboard/quicksight/yaml.md +++ b/openmetadata-docs/content/v1.7.x-SNAPSHOT/connectors/dashboard/quicksight/yaml.md @@ -103,16 +103,7 @@ This is a sample config for QuickSight: #### Source Configuration - Service Connection -{% codeInfo srNumber=1 %} - -**awsConfig** - - **AWS Access Key ID**: Enter your secure access key ID for your Glue connection. The specified key ID should be authorized to read all databases you want to include in the metadata ingestion workflow. - - **AWS Secret Access Key**: Enter the Secret Access Key (the passcode key pair to the key ID from above). - - **AWS Region**: Enter the location of the amazon cluster that your data and account are associated with. - - **AWS Session Token (optional)**: The AWS session token is an optional parameter. If you want, enter the details of your temporary session token. - - **Endpoint URL (optional)**: Your Glue connector will automatically determine the AWS QuickSight endpoint URL based on the region. You may override this behavior by entering a value to the endpoint URL. - -{% /codeInfo %} +{% partial file="/v1.7/connectors/yaml/common/aws-config-def.md" /%} {% codeInfo srNumber=2 %} @@ -150,13 +141,9 @@ source: config: type: QuickSight ``` -```yaml {% srNumber=1 %} - awsConfig: - awsAccessKeyId: KEY - awsSecretAccessKey: SECRET - awsRegion: us-east-2 - awsSessionToken: Token -``` + +{% partial file="/v1.7/connectors/yaml/common/aws-config.md" /%} + ```yaml {% srNumber=2 %} awsAccountId: ``` diff --git a/openmetadata-docs/content/v1.7.x-SNAPSHOT/connectors/database/athena/yaml.md b/openmetadata-docs/content/v1.7.x-SNAPSHOT/connectors/database/athena/yaml.md index 125ea35f87a..3055f461510 100644 --- a/openmetadata-docs/content/v1.7.x-SNAPSHOT/connectors/database/athena/yaml.md +++ b/openmetadata-docs/content/v1.7.x-SNAPSHOT/connectors/database/athena/yaml.md @@ -172,100 +172,7 @@ This is a sample config for Athena: #### Source Configuration - Service Connection -{% codeInfo srNumber=1 %} - -- **awsAccessKeyId** & **awsSecretAccessKey**: When you interact with AWS, you specify your AWS security credentials to verify who you are and whether you have - permission to access the resources that you are requesting. AWS uses the security credentials to authenticate and - authorize your requests ([docs](https://docs.aws.amazon.com/IAM/latest/UserGuide/security-creds.html)). - -Access keys consist of two parts: An **access key ID** (for example, `AKIAIOSFODNN7EXAMPLE`), and a **secret access key** (for example, `wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY`). - -You must use both the access key ID and secret access key together to authenticate your requests. - -You can find further information on how to manage your access keys [here](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html). - -{% /codeInfo %} - -{% codeInfo srNumber=2 %} -**awsSessionToken**: If you are using temporary credentials to access your services, you will need to inform the AWS Access Key ID -and AWS Secrets Access Key. Also, these will include an AWS Session Token. - -{% /codeInfo %} - -{% codeInfo srNumber=3 %} - -**awsRegion**: Each AWS Region is a separate geographic area in which AWS clusters data centers ([docs](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.RegionsAndAvailabilityZones.html)). - -As AWS can have instances in multiple regions, we need to know the region the service you want reach belongs to. - -Note that the AWS Region is the only required parameter when configuring a connection. When connecting to the -services programmatically, there are different ways in which we can extract and use the rest of AWS configurations. - -You can find further information about configuring your credentials [here](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html#configuring-credentials). - -{% /codeInfo %} - -{% codeInfo srNumber=4 %} - -**endPointURL**: To connect programmatically to an AWS service, you use an endpoint. An *endpoint* is the URL of the -entry point for an AWS web service. The AWS SDKs and the AWS Command Line Interface (AWS CLI) automatically use the -default endpoint for each service in an AWS Region. But you can specify an alternate endpoint for your API requests. - -Find more information on [AWS service endpoints](https://docs.aws.amazon.com/general/latest/gr/rande.html). - -{% /codeInfo %} - -{% codeInfo srNumber=5 %} - -**profileName**: A named profile is a collection of settings and credentials that you can apply to a AWS CLI command. -When you specify a profile to run a command, the settings and credentials are used to run that command. -Multiple named profiles can be stored in the config and credentials files. - -You can inform this field if you'd like to use a profile other than `default`. - -Find here more information about [Named profiles for the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-profiles.html). - -{% /codeInfo %} - -{% codeInfo srNumber=6 %} - -**assumeRoleArn**: Typically, you use `AssumeRole` within your account or for cross-account access. In this field you'll set the -`ARN` (Amazon Resource Name) of the policy of the other account. - -A user who wants to access a role in a different account must also have permissions that are delegated from the account -administrator. The administrator must attach a policy that allows the user to call `AssumeRole` for the `ARN` of the role in the other account. - -This is a required field if you'd like to `AssumeRole`. - -Find more information on [AssumeRole](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html). - -{%note%} -When using Assume Role authentication, ensure you provide the following details: -- **AWS Region**: Specify the AWS region for your deployment. -- **Assume Role ARN**: Provide the ARN of the role in your AWS account that OpenMetadata will assume. -{%/note%} - -{% /codeInfo %} - -{% codeInfo srNumber=7 %} - -**assumeRoleSessionName**: An identifier for the assumed role session. Use the role session name to uniquely identify a session when the same role -is assumed by different principals or for different reasons. - -By default, we'll use the name `OpenMetadataSession`. - -Find more information about the [Role Session Name](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html#:~:text=An%20identifier%20for%20the%20assumed%20role%20session.). - -{% /codeInfo %} - -{% codeInfo srNumber=8 %} - -**assumeRoleSourceIdentity**: The source identity specified by the principal that is calling the `AssumeRole` operation. You can use source identity -information in AWS CloudTrail logs to determine who took actions with a role. - -Find more information about [Source Identity](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html#:~:text=Required%3A%20No-,SourceIdentity,-The%20source%20identity). - -{% /codeInfo %} +{% partial file="/v1.7/connectors/yaml/common/aws-config-def.md" /%} {% codeInfo srNumber=9 %} @@ -312,31 +219,9 @@ source: type: Athena awsConfig: ``` -```yaml {% srNumber=1 %} - awsAccessKeyId: KEY - awsSecretAccessKey: SECRET -``` -```yaml {% srNumber=2 %} - # awsSessionToken: TOKEN -``` -```yaml {% srNumber=3 %} - awsRegion: us-east-2 -``` -```yaml {% srNumber=4 %} - # endPointURL: https://athena.us-east-2.amazonaws.com/custom -``` -```yaml {% srNumber=5 %} - # profileName: profile -``` -```yaml {% srNumber=6 %} - # assumeRoleArn: "arn:partition:service:region:account:resource" -``` -```yaml {% srNumber=7 %} - # assumeRoleSessionName: session -``` -```yaml {% srNumber=8 %} - # assumeRoleSourceIdentity: identity -``` + +{% partial file="/v1.7/connectors/yaml/common/aws-config.md" /%} + ```yaml {% srNumber=9 %} s3StagingDir: s3 directory for datasource ``` diff --git a/openmetadata-docs/content/v1.7.x-SNAPSHOT/connectors/database/dynamodb/yaml.md b/openmetadata-docs/content/v1.7.x-SNAPSHOT/connectors/database/dynamodb/yaml.md index 69bbde6dc1c..dca9e9b775a 100644 --- a/openmetadata-docs/content/v1.7.x-SNAPSHOT/connectors/database/dynamodb/yaml.md +++ b/openmetadata-docs/content/v1.7.x-SNAPSHOT/connectors/database/dynamodb/yaml.md @@ -78,35 +78,7 @@ This is a sample config for DynamoDB: #### Source Configuration - Service Connection -{% codeInfo srNumber=1 %} - -**awsAccessKeyId**: Enter your secure access key ID for your DynamoDB connection. The specified key ID should be authorized to read all databases you want to include in the metadata ingestion workflow. - -{% /codeInfo %} - -{% codeInfo srNumber=2 %} - -**awsSecretAccessKey**: Enter the Secret Access Key (the passcode key pair to the key ID from above). - -{% /codeInfo %} - -{% codeInfo srNumber=3 %} - -**awsSessionToken**: The AWS session token is an optional parameter. If you want, enter the details of your temporary session token. - -{% /codeInfo %} - -{% codeInfo srNumber=4 %} - -**awsRegion**: Enter the location of the amazon cluster that your data and account are associated with. - -{% /codeInfo %} - -{% codeInfo srNumber=5 %} - -**endPointURL**: Your DynamoDB connector will automatically determine the AWS DynamoDB endpoint URL based on the region. You may override this behavior by entering a value to the endpoint URL. - -{% /codeInfo %} +{% partial file="/v1.7/connectors/yaml/common/aws-config-def.md" /%} {% codeInfo srNumber=6 %} @@ -149,21 +121,9 @@ source: type: DynamoDB awsConfig: ``` -```yaml {% srNumber=1 %} - awsAccessKeyId: aws_access_key_id -``` -```yaml {% srNumber=2 %} - awsSecretAccessKey: aws_secret_access_key -``` -```yaml {% srNumber=3 %} - awsSessionToken: AWS Session Token -``` -```yaml {% srNumber=4 %} - awsRegion: aws region -``` -```yaml {% srNumber=5 %} - endPointURL: https://dynamodb..amazonaws.com -``` + +{% partial file="/v1.7/connectors/yaml/common/aws-config.md" /%} + ```yaml {% srNumber=6 %} database: custom_database_name ``` diff --git a/openmetadata-docs/content/v1.7.x-SNAPSHOT/connectors/database/glue/yaml.md b/openmetadata-docs/content/v1.7.x-SNAPSHOT/connectors/database/glue/yaml.md index 88b63d8a557..305afd5b408 100644 --- a/openmetadata-docs/content/v1.7.x-SNAPSHOT/connectors/database/glue/yaml.md +++ b/openmetadata-docs/content/v1.7.x-SNAPSHOT/connectors/database/glue/yaml.md @@ -58,35 +58,7 @@ This is a sample config for Glue: #### Source Configuration - Service Connection -{% codeInfo srNumber=1 %} - -**awsAccessKeyId**: Enter your secure access key ID for your Glue connection. The specified key ID should be authorized to read all databases you want to include in the metadata ingestion workflow. - -{% /codeInfo %} - -{% codeInfo srNumber=2 %} - -**awsSecretAccessKey**: Enter the Secret Access Key (the passcode key pair to the key ID from above). - -{% /codeInfo %} - -{% codeInfo srNumber=3 %} - -**awsRegion**: Enter the location of the amazon cluster that your data and account are associated with. - -{% /codeInfo %} - -{% codeInfo srNumber=4 %} - -**endPointURL**: Your Glue connector will automatically determine the AWS Glue endpoint URL based on the region. You may override this behavior by entering a value to the endpoint URL. - -{% /codeInfo %} - -{% codeInfo srNumber=5 %} - -**awsSessionToken**: The AWS session token is an optional parameter. If you want, enter the details of your temporary session token. - -{% /codeInfo %} +{% partial file="/v1.7/connectors/yaml/common/aws-config-def.md" /%} {% codeInfo srNumber=6 %} @@ -130,21 +102,9 @@ source: type: Glue awsConfig: ``` -```yaml {% srNumber=1 %} - awsAccessKeyId: KEY -``` -```yaml {% srNumber=2 %} - awsSecretAccessKey: SECRET -``` -```yaml {% srNumber=3 %} - awsRegion: us-east-2 -``` -```yaml {% srNumber=4 %} - # endPointURL: https://glue.us-east-2.amazonaws.com/ -``` -```yaml {% srNumber=5 %} - # awsSessionToken: TOKEN -``` + +{% partial file="/v1.7/connectors/yaml/common/aws-config.md" /%} + ```yaml {% srNumber=6 %} databaseName: database_name ``` diff --git a/openmetadata-docs/content/v1.7.x-SNAPSHOT/connectors/database/s3-datalake/yaml.md b/openmetadata-docs/content/v1.7.x-SNAPSHOT/connectors/database/s3-datalake/yaml.md index c2f39ba8a1c..4654c40ceb3 100644 --- a/openmetadata-docs/content/v1.7.x-SNAPSHOT/connectors/database/s3-datalake/yaml.md +++ b/openmetadata-docs/content/v1.7.x-SNAPSHOT/connectors/database/s3-datalake/yaml.md @@ -90,13 +90,11 @@ The workflow is modeled around the following JSON Schema. {% codeInfo srNumber=1 %} -* **awsAccessKeyId**: Enter your secure access key ID for your DynamoDB connection. The specified key ID should be authorized to read all databases you want to include in the metadata ingestion workflow. -* **awsSecretAccessKey**: Enter the Secret Access Key (the passcode key pair to the key ID from above). -* **awsRegion**: Specify the region in which your DynamoDB is located. This setting is required even if you have configured a local AWS profile. * **schemaFilterPattern** and **tableFilterPattern**: Note that the `schemaFilterPattern` and `tableFilterPattern` both support regex as `include` or `exclude`. E.g., {% /codeInfo %} +{% partial file="/v1.7/connectors/yaml/common/aws-config-def.md" /%} {% partial file="/v1.7/connectors/yaml/database/source-config-def.md" /%} @@ -120,9 +118,11 @@ source: ```yaml {% srNumber=1 %} configSource: securityConfig: - awsAccessKeyId: aws access key id - awsSecretAccessKey: aws secret access key - awsRegion: aws region +``` + +{% partial file="/v1.7/connectors/yaml/common/aws-config.md" /%} + +```yaml {% srNumber=1 %} bucketName: bucket name prefix: prefix ``` diff --git a/openmetadata-docs/content/v1.7.x-SNAPSHOT/connectors/ingestion/workflows/dbt/ingest-dbt-yaml.md b/openmetadata-docs/content/v1.7.x-SNAPSHOT/connectors/ingestion/workflows/dbt/ingest-dbt-yaml.md index 5c6b7950758..674c37d8c47 100644 --- a/openmetadata-docs/content/v1.7.x-SNAPSHOT/connectors/ingestion/workflows/dbt/ingest-dbt-yaml.md +++ b/openmetadata-docs/content/v1.7.x-SNAPSHOT/connectors/ingestion/workflows/dbt/ingest-dbt-yaml.md @@ -46,38 +46,7 @@ In this configuration we will be fetching the dbt `manifest.json`, `catalog.json - **dbtConfigType**: s3 {% /codeInfo %} -{% codeInfo srNumber=1 %} - -- **awsAccessKeyId** & **awsSecretAccessKey**: When you interact with AWS, you specify your AWS security credentials to verify who you are and whether you have - permission to access the resources that you are requesting. AWS uses the security credentials to authenticate and - authorize your requests ([docs](https://docs.aws.amazon.com/IAM/latest/UserGuide/security-creds.html)). - -Access keys consist of two parts: An **access key ID** (for example, `AKIAIOSFODNN7EXAMPLE`), and a **secret access key** (for example, `wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY`). - -You must use both the access key ID and secret access key together to authenticate your requests. - -You can find further information on how to manage your access keys [here](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html). - -{% /codeInfo %} - -{% codeInfo srNumber=2 %} -**awsSessionToken**: If you are using temporary credentials to access your services, you will need to inform the AWS Access Key ID -and AWS Secrets Access Key. Also, these will include an AWS Session Token. - -{% /codeInfo %} - -{% codeInfo srNumber=3 %} - -**awsRegion**: Each AWS Region is a separate geographic area in which AWS clusters data centers ([docs](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.RegionsAndAvailabilityZones.html)). - -As AWS can have instances in multiple regions, we need to know the region the service you want reach belongs to. - -Note that the AWS Region is the only required parameter when configuring a connection. When connecting to the -services programmatically, there are different ways in which we can extract and use the rest of AWS configurations. - -You can find further information about configuring your credentials [here](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html#configuring-credentials). - -{% /codeInfo %} +{% partial file="/v1.7/connectors/yaml/common/aws-config-def.md" /%} {% partial file="/v1.7/connectors/yaml/dbt/dbt-prefix-def.md" /%} @@ -105,16 +74,10 @@ source: ``` ```yaml {% srNumber=1 %} dbtSecurityConfig: - awsAccessKeyId: AWS_ACCESS_KEY_ID - awsSecretAccessKey: AWS_SECRET_ACCESS_KEY -``` -```yaml {% srNumber=2 %} - awsSessionToken: AWS_SESSION_TOKEN -``` -```yaml {% srNumber=3 %} - awsRegion: AWS_REGION ``` +{% partial file="/v1.7/connectors/yaml/common/aws-config.md" /%} + {% partial file="/v1.7/connectors/yaml/dbt/dbt-prefix.md" /%} {% partial file="/v1.7/connectors/yaml/dbt/source-config.md" /%} diff --git a/openmetadata-docs/content/v1.7.x-SNAPSHOT/connectors/messaging/kinesis/yaml.md b/openmetadata-docs/content/v1.7.x-SNAPSHOT/connectors/messaging/kinesis/yaml.md index f15cf02cc13..66cba6c4f59 100644 --- a/openmetadata-docs/content/v1.7.x-SNAPSHOT/connectors/messaging/kinesis/yaml.md +++ b/openmetadata-docs/content/v1.7.x-SNAPSHOT/connectors/messaging/kinesis/yaml.md @@ -80,100 +80,7 @@ This is a sample config for Kinesis: #### Source Configuration - Service Connection -{% codeInfo srNumber=1 %} - -- **awsAccessKeyId** & **awsSecretAccessKey**: When you interact with AWS, you specify your AWS security credentials to verify who you are and whether you have - permission to access the resources that you are requesting. AWS uses the security credentials to authenticate and - authorize your requests ([docs](https://docs.aws.amazon.com/IAM/latest/UserGuide/security-creds.html)). - -Access keys consist of two parts: An **access key ID** (for example, `AKIAIOSFODNN7EXAMPLE`), and a **secret access key** (for example, `wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY`). - -You must use both the access key ID and secret access key together to authenticate your requests. - -You can find further information on how to manage your access keys [here](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html). - -{% /codeInfo %} - -{% codeInfo srNumber=2 %} -**awsSessionToken**: If you are using temporary credentials to access your services, you will need to inform the AWS Access Key ID -and AWS Secrets Access Key. Also, these will include an AWS Session Token. - -{% /codeInfo %} - -{% codeInfo srNumber=3 %} - -**awsRegion**: Each AWS Region is a separate geographic area in which AWS clusters data centers ([docs](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.RegionsAndAvailabilityZones.html)). - -As AWS can have instances in multiple regions, we need to know the region the service you want reach belongs to. - -Note that the AWS Region is the only required parameter when configuring a connection. When connecting to the -services programmatically, there are different ways in which we can extract and use the rest of AWS configurations. - -You can find further information about configuring your credentials [here](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html#configuring-credentials). - -{% /codeInfo %} - -{% codeInfo srNumber=4 %} - -**endPointURL**: To connect programmatically to an AWS service, you use an endpoint. An *endpoint* is the URL of the -entry point for an AWS web service. The AWS SDKs and the AWS Command Line Interface (AWS CLI) automatically use the -default endpoint for each service in an AWS Region. But you can specify an alternate endpoint for your API requests. - -Find more information on [AWS service endpoints](https://docs.aws.amazon.com/general/latest/gr/rande.html). - -{% /codeInfo %} - -{% codeInfo srNumber=5 %} - -**profileName**: A named profile is a collection of settings and credentials that you can apply to a AWS CLI command. -When you specify a profile to run a command, the settings and credentials are used to run that command. -Multiple named profiles can be stored in the config and credentials files. - -You can inform this field if you'd like to use a profile other than `default`. - -Find here more information about [Named profiles for the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-profiles.html). - -{% /codeInfo %} - -{% codeInfo srNumber=6 %} - -**assumeRoleArn**: Typically, you use `AssumeRole` within your account or for cross-account access. In this field you'll set the -`ARN` (Amazon Resource Name) of the policy of the other account. - -A user who wants to access a role in a different account must also have permissions that are delegated from the account -administrator. The administrator must attach a policy that allows the user to call `AssumeRole` for the `ARN` of the role in the other account. - -This is a required field if you'd like to `AssumeRole`. - -Find more information on [AssumeRole](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html). - -{%note%} -When using Assume Role authentication, ensure you provide the following details: -- **AWS Region**: Specify the AWS region for your deployment. -- **Assume Role ARN**: Provide the ARN of the role in your AWS account that OpenMetadata will assume. -{%/note%} - -{% /codeInfo %} - -{% codeInfo srNumber=7 %} - -**assumeRoleSessionName**: An identifier for the assumed role session. Use the role session name to uniquely identify a session when the same role -is assumed by different principals or for different reasons. - -By default, we'll use the name `OpenMetadataSession`. - -Find more information about the [Role Session Name](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html#:~:text=An%20identifier%20for%20the%20assumed%20role%20session.). - -{% /codeInfo %} - -{% codeInfo srNumber=8 %} - -**assumeRoleSourceIdentity**: The source identity specified by the principal that is calling the `AssumeRole` operation. You can use source identity -information in AWS CloudTrail logs to determine who took actions with a role. - -Find more information about [Source Identity](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html#:~:text=Required%3A%20No-,SourceIdentity,-The%20source%20identity). - -{% /codeInfo %} +{% partial file="/v1.7/connectors/yaml/common/aws-config-def.md" /%} {% partial file="/v1.7/connectors/yaml/messaging/source-config-def.md" /%} @@ -195,31 +102,8 @@ source: type: Kinesis awsConfig: ``` -```yaml {% srNumber=1 %} - awsAccessKeyId: KEY - awsSecretAccessKey: SECRET -``` -```yaml {% srNumber=2 %} - # awsSessionToken: TOKEN -``` -```yaml {% srNumber=3 %} - awsRegion: us-east-2 -``` -```yaml {% srNumber=4 %} - # endPointURL: https://athena.us-east-2.amazonaws.com/custom -``` -```yaml {% srNumber=5 %} - # profileName: profile -``` -```yaml {% srNumber=6 %} - # assumeRoleArn: "arn:partition:service:region:account:resource" -``` -```yaml {% srNumber=7 %} - # assumeRoleSessionName: session -``` -```yaml {% srNumber=8 %} - # assumeRoleSourceIdentity: identity -``` + +{% partial file="/v1.7/connectors/yaml/common/aws-config.md" /%} {% partial file="/v1.7/connectors/yaml/messaging/source-config.md" /%} diff --git a/openmetadata-docs/content/v1.7.x-SNAPSHOT/connectors/ml-model/sagemaker/yaml.md b/openmetadata-docs/content/v1.7.x-SNAPSHOT/connectors/ml-model/sagemaker/yaml.md index bb5d4365540..f4006cd42f7 100644 --- a/openmetadata-docs/content/v1.7.x-SNAPSHOT/connectors/ml-model/sagemaker/yaml.md +++ b/openmetadata-docs/content/v1.7.x-SNAPSHOT/connectors/ml-model/sagemaker/yaml.md @@ -78,100 +78,7 @@ This is a sample config for Sagemaker: #### Source Configuration - Service Connection -{% codeInfo srNumber=1 %} - -- **awsAccessKeyId** & **awsSecretAccessKey**: When you interact with AWS, you specify your AWS security credentials to verify who you are and whether you have - permission to access the resources that you are requesting. AWS uses the security credentials to authenticate and - authorize your requests ([docs](https://docs.aws.amazon.com/IAM/latest/UserGuide/security-creds.html)). - -Access keys consist of two parts: An **access key ID** (for example, `AKIAIOSFODNN7EXAMPLE`), and a **secret access key** (for example, `wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY`). - -You must use both the access key ID and secret access key together to authenticate your requests. - -You can find further information on how to manage your access keys [here](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html). - -{% /codeInfo %} - -{% codeInfo srNumber=2 %} -**awsSessionToken**: If you are using temporary credentials to access your services, you will need to inform the AWS Access Key ID -and AWS Secrets Access Key. Also, these will include an AWS Session Token. - -{% /codeInfo %} - -{% codeInfo srNumber=3 %} - -**awsRegion**: Each AWS Region is a separate geographic area in which AWS clusters data centers ([docs](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.RegionsAndAvailabilityZones.html)). - -As AWS can have instances in multiple regions, we need to know the region the service you want reach belongs to. - -Note that the AWS Region is the only required parameter when configuring a connection. When connecting to the -services programmatically, there are different ways in which we can extract and use the rest of AWS configurations. - -You can find further information about configuring your credentials [here](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html#configuring-credentials). - -{% /codeInfo %} - -{% codeInfo srNumber=4 %} - -**endPointURL**: To connect programmatically to an AWS service, you use an endpoint. An *endpoint* is the URL of the -entry point for an AWS web service. The AWS SDKs and the AWS Command Line Interface (AWS CLI) automatically use the -default endpoint for each service in an AWS Region. But you can specify an alternate endpoint for your API requests. - -Find more information on [AWS service endpoints](https://docs.aws.amazon.com/general/latest/gr/rande.html). - -{% /codeInfo %} - -{% codeInfo srNumber=5 %} - -**profileName**: A named profile is a collection of settings and credentials that you can apply to a AWS CLI command. -When you specify a profile to run a command, the settings and credentials are used to run that command. -Multiple named profiles can be stored in the config and credentials files. - -You can inform this field if you'd like to use a profile other than `default`. - -Find here more information about [Named profiles for the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-profiles.html). - -{% /codeInfo %} - -{% codeInfo srNumber=6 %} - -**assumeRoleArn**: Typically, you use `AssumeRole` within your account or for cross-account access. In this field you'll set the -`ARN` (Amazon Resource Name) of the policy of the other account. - -A user who wants to access a role in a different account must also have permissions that are delegated from the account -administrator. The administrator must attach a policy that allows the user to call `AssumeRole` for the `ARN` of the role in the other account. - -This is a required field if you'd like to `AssumeRole`. - -Find more information on [AssumeRole](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html). - -{%note%} -When using Assume Role authentication, ensure you provide the following details: -- **AWS Region**: Specify the AWS region for your deployment. -- **Assume Role ARN**: Provide the ARN of the role in your AWS account that OpenMetadata will assume. -{%/note%} - -{% /codeInfo %} - -{% codeInfo srNumber=7 %} - -**assumeRoleSessionName**: An identifier for the assumed role session. Use the role session name to uniquely identify a session when the same role -is assumed by different principals or for different reasons. - -By default, we'll use the name `OpenMetadataSession`. - -Find more information about the [Role Session Name](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html#:~:text=An%20identifier%20for%20the%20assumed%20role%20session.). - -{% /codeInfo %} - -{% codeInfo srNumber=8 %} - -**assumeRoleSourceIdentity**: The source identity specified by the principal that is calling the `AssumeRole` operation. You can use source identity -information in AWS CloudTrail logs to determine who took actions with a role. - -Find more information about [Source Identity](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html#:~:text=Required%3A%20No-,SourceIdentity,-The%20source%20identity). - -{% /codeInfo %} +{% partial file="/v1.7/connectors/yaml/common/aws-config-def.md" /%} {% partial file="/v1.7/connectors/yaml/ml-model/source-config-def.md" /%} @@ -193,31 +100,8 @@ source: type: SageMaker awsConfig: ``` -```yaml {% srNumber=1 %} - awsAccessKeyId: KEY - awsSecretAccessKey: SECRET -``` -```yaml {% srNumber=2 %} - # awsSessionToken: TOKEN -``` -```yaml {% srNumber=3 %} - awsRegion: us-east-2 -``` -```yaml {% srNumber=4 %} - # endPointURL: https://athena.us-east-2.amazonaws.com/custom -``` -```yaml {% srNumber=5 %} - # profileName: profile -``` -```yaml {% srNumber=6 %} - # assumeRoleArn: "arn:partition:service:region:account:resource" -``` -```yaml {% srNumber=7 %} - # assumeRoleSessionName: session -``` -```yaml {% srNumber=8 %} - # assumeRoleSourceIdentity: identity -``` + +{% partial file="/v1.7/connectors/yaml/common/aws-config.md" /%} {% partial file="/v1.7/connectors/yaml/ml-model/source-config.md" /%} diff --git a/openmetadata-docs/content/v1.7.x-SNAPSHOT/connectors/pipeline/glue-pipeline/yaml.md b/openmetadata-docs/content/v1.7.x-SNAPSHOT/connectors/pipeline/glue-pipeline/yaml.md index 4c3ff391c3a..3dcd30307fd 100644 --- a/openmetadata-docs/content/v1.7.x-SNAPSHOT/connectors/pipeline/glue-pipeline/yaml.md +++ b/openmetadata-docs/content/v1.7.x-SNAPSHOT/connectors/pipeline/glue-pipeline/yaml.md @@ -63,58 +63,7 @@ This is a sample config for Glue: #### Source Configuration - Service Connection -{% codeInfo srNumber=1 %} - - -**awsAccessKeyId**: Enter your secure access key ID for your Glue connection. The specified key ID should be - authorized to read all databases you want to include in the metadata ingestion workflow. - -{% /codeInfo %} - -{% codeInfo srNumber=2 %} - -**awsSecretAccessKey**: Enter the Secret Access Key (the passcode key pair to the key ID from above). - - -{% /codeInfo %} - -{% codeInfo srNumber=3 %} - -**awsRegion**: Enter the location of the amazon cluster that your data and account are associated with. - -{% /codeInfo %} - -{% codeInfo srNumber=4 %} - -**awsSessionToken**: The AWS session token is an optional parameter. If you want, enter the details of your temporary - session token. - -{% /codeInfo %} - -{% codeInfo srNumber=5 %} - -**endPointURL**: Your Glue connector will automatically determine the AWS Glue endpoint URL based on the region. You - may override this behavior by entering a value to the endpoint URL. - - -{% /codeInfo %} - -{% codeInfo srNumber=6 %} -**profileName**: The name of a profile to use with the boto session. -{% /codeInfo %} - -{% codeInfo srNumber=7 %} -**assumeRoleArn**: The Amazon Resource Name (ARN) of the role to assume. Required Field in case of Assume Role. -{% /codeInfo %} - -{% codeInfo srNumber=8 %} -**assumeRoleSessionName**: An identifier for the assumed role session. Use the role session name to uniquely identify a session when the same role is assumed by different principals or for different reasons. Required Field in case of Assume Role. -{% /codeInfo %} - -{% codeInfo srNumber=9 %} -**assumeRoleSourceIdentity**: The Amazon Resource Name (ARN) of the role to assume. Optional Field in case of Assume Role. -{% /codeInfo %} - +{% partial file="/v1.7/connectors/yaml/common/aws-config-def.md" /%} {% partial file="/v1.6/connectors/yaml/pipeline/source-config-def.md" /%} @@ -135,34 +84,7 @@ source: type: GluePipeline awsConfig: ``` -```yaml {% srNumber=1 %} - awsAccessKeyId: KEY -``` -```yaml {% srNumber=2 %} - awsSecretAccessKey: SECRET -``` -```yaml {% srNumber=3 %} - awsRegion: us-east-2 -``` -```yaml {% srNumber=4 %} - # awsSessionToken: TOKEN -``` -```yaml {% srNumber=5 %} - # endPointURL: https://glue.us-east-2.amazonaws.com/ -``` -```yaml {% srNumber=6 %} - # profileName: "" -``` -```yaml {% srNumber=7 %} - # assumeRoleArn: "" -``` -```yaml {% srNumber=8 %} - # assumeRoleSessionName: OpenMetadataSession -``` -```yaml {% srNumber=9 %} - # assumeRoleSourceIdentity : "" -``` - +{% partial file="/v1.7/connectors/yaml/common/aws-config.md" /%} {% partial file="/v1.6/connectors/yaml/pipeline/source-config.md" /%} diff --git a/openmetadata-docs/content/v1.7.x-SNAPSHOT/connectors/storage/s3/yaml.md b/openmetadata-docs/content/v1.7.x-SNAPSHOT/connectors/storage/s3/yaml.md index ac2f7511ef6..5c8b88c4045 100644 --- a/openmetadata-docs/content/v1.7.x-SNAPSHOT/connectors/storage/s3/yaml.md +++ b/openmetadata-docs/content/v1.7.x-SNAPSHOT/connectors/storage/s3/yaml.md @@ -128,100 +128,7 @@ This is a sample config for Athena: #### Source Configuration - Service Connection -{% codeInfo srNumber=1 %} - -- **awsAccessKeyId** & **awsSecretAccessKey**: When you interact with AWS, you specify your AWS security credentials to verify who you are and whether you have - permission to access the resources that you are requesting. AWS uses the security credentials to authenticate and - authorize your requests ([docs](https://docs.aws.amazon.com/IAM/latest/UserGuide/security-creds.html)). - -Access keys consist of two parts: An **access key ID** (for example, `AKIAIOSFODNN7EXAMPLE`), and a **secret access key** (for example, `wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY`). - -You must use both the access key ID and secret access key together to authenticate your requests. - -You can find further information on how to manage your access keys [here](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html). - -{% /codeInfo %} - -{% codeInfo srNumber=2 %} -**awsSessionToken**: If you are using temporary credentials to access your services, you will need to inform the AWS Access Key ID -and AWS Secrets Access Key. Also, these will include an AWS Session Token. - -{% /codeInfo %} - -{% codeInfo srNumber=3 %} - -**awsRegion**: Each AWS Region is a separate geographic area in which AWS clusters data centers ([docs](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.RegionsAndAvailabilityZones.html)). - -As AWS can have instances in multiple regions, we need to know the region the service you want reach belongs to. - -Note that the AWS Region is the only required parameter when configuring a connection. When connecting to the -services programmatically, there are different ways in which we can extract and use the rest of AWS configurations. - -You can find further information about configuring your credentials [here](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html#configuring-credentials). - -{% /codeInfo %} - -{% codeInfo srNumber=4 %} - -**endPointURL**: To connect programmatically to an AWS service, you use an endpoint. An *endpoint* is the URL of the -entry point for an AWS web service. The AWS SDKs and the AWS Command Line Interface (AWS CLI) automatically use the -default endpoint for each service in an AWS Region. But you can specify an alternate endpoint for your API requests. - -Find more information on [AWS service endpoints](https://docs.aws.amazon.com/general/latest/gr/rande.html). - -{% /codeInfo %} - -{% codeInfo srNumber=5 %} - -**profileName**: A named profile is a collection of settings and credentials that you can apply to a AWS CLI command. -When you specify a profile to run a command, the settings and credentials are used to run that command. -Multiple named profiles can be stored in the config and credentials files. - -You can inform this field if you'd like to use a profile other than `default`. - -Find here more information about [Named profiles for the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-profiles.html). - -{% /codeInfo %} - -{% codeInfo srNumber=6 %} - -**assumeRoleArn**: Typically, you use `AssumeRole` within your account or for cross-account access. In this field you'll set the -`ARN` (Amazon Resource Name) of the policy of the other account. - -A user who wants to access a role in a different account must also have permissions that are delegated from the account -administrator. The administrator must attach a policy that allows the user to call `AssumeRole` for the `ARN` of the role in the other account. - -This is a required field if you'd like to `AssumeRole`. - -Find more information on [AssumeRole](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html). - -{%note%} -When using Assume Role authentication, ensure you provide the following details: -- **AWS Region**: Specify the AWS region for your deployment. -- **Assume Role ARN**: Provide the ARN of the role in your AWS account that OpenMetadata will assume. -{%/note%} - -{% /codeInfo %} - -{% codeInfo srNumber=7 %} - -**assumeRoleSessionName**: An identifier for the assumed role session. Use the role session name to uniquely identify a session when the same role -is assumed by different principals or for different reasons. - -By default, we'll use the name `OpenMetadataSession`. - -Find more information about the [Role Session Name](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html#:~:text=An%20identifier%20for%20the%20assumed%20role%20session.). - -{% /codeInfo %} - -{% codeInfo srNumber=8 %} - -**assumeRoleSourceIdentity**: The source identity specified by the principal that is calling the `AssumeRole` operation. You can use source identity -information in AWS CloudTrail logs to determine who took actions with a role. - -Find more information about [Source Identity](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html#:~:text=Required%3A%20No-,SourceIdentity,-The%20source%20identity). - -{% /codeInfo %} +{% partial file="/v1.7/connectors/yaml/common/aws-config-def.md" /%} {% partial file="/v1.7/connectors/yaml/storage/source-config-def.md" /%} @@ -262,31 +169,9 @@ source: type: S3 awsConfig: ``` -```yaml {% srNumber=1 %} - awsAccessKeyId: KEY - awsSecretAccessKey: SECRET -``` -```yaml {% srNumber=2 %} - # awsSessionToken: TOKEN -``` -```yaml {% srNumber=3 %} - awsRegion: us-east-2 -``` -```yaml {% srNumber=4 %} - # endPointURL: https://s3.us-east-2.amazonaws.com/custom -``` -```yaml {% srNumber=5 %} - # profileName: profile -``` -```yaml {% srNumber=6 %} - # assumeRoleArn: "arn:partition:service:region:account:resource" -``` -```yaml {% srNumber=7 %} - # assumeRoleSessionName: session -``` -```yaml {% srNumber=8 %} - # assumeRoleSourceIdentity: identity -``` + +{% partial file="/v1.7/connectors/yaml/common/aws-config.md" /%} + ```yaml {% srNumber=13 %} bucketNames: - s3-testing-1