mirror of
https://github.com/open-metadata/OpenMetadata.git
synced 2025-07-13 20:18:24 +00:00
114 lines
4.0 KiB
Markdown
114 lines
4.0 KiB
Markdown
|
---
|
||
|
|
title: AWS Systems Manager Parameter Store
|
||
|
|
slug: /deployment/secrets-manager/supported-implementations/aws-ssm-parameter-store
|
||
|
|
---
|
||
|
|
|
||
|
|
# AWS Systems Manager Parameter Store
|
||
|
|
|
||
|
|
The setup steps covers the use of the managed version of the AWS Systems Manager Parameter Store as secrets manager but
|
||
|
|
for the non-managed follow only the steps related to the Airflow server and CLI.
|
||
|
|
|
||
|
|
## Setup
|
||
|
|
|
||
|
|
### 1. Permissions needed
|
||
|
|
|
||
|
|
These are the permissions required in the IAM policy to enable the AWS Systems Manager Parameter Store in OpenMetadata.
|
||
|
|
|
||
|
|
```json
|
||
|
|
{
|
||
|
|
"Version": "2012-10-17",
|
||
|
|
"Statement": [
|
||
|
|
{
|
||
|
|
"Effect": "Allow",
|
||
|
|
"Action": [
|
||
|
|
"ssm:PutParameter",
|
||
|
|
"ssm:GetParameter"
|
||
|
|
],
|
||
|
|
"Resource": "*"
|
||
|
|
}
|
||
|
|
]
|
||
|
|
}
|
||
|
|
```
|
||
|
|
|
||
|
|
### 2. Update configuration
|
||
|
|
|
||
|
|
We have to set up the secret manager provider we want to use, that in our case is `aws-ssm`, and the credentials for our
|
||
|
|
AWS account.
|
||
|
|
|
||
|
|
The changes to be done in `openmetadata.yaml` file of the OpenMetadata server are:
|
||
|
|
|
||
|
|
```yaml
|
||
|
|
...
|
||
|
|
secretsManagerConfiguration:
|
||
|
|
secretsManager: managed-aws-ssm # or env var SECRET_MANAGER. For non-managed use 'aws-ssm'.
|
||
|
|
parameters:
|
||
|
|
region: <aws region> # or env var OM_SM_REGION
|
||
|
|
accessKeyId: <aws access key id> # or env var OM_SM_ACCESS_KEY_ID
|
||
|
|
secretAccessKey: <aws secret access key> # or env var OM_SM_ACCESS_KEY
|
||
|
|
...
|
||
|
|
```
|
||
|
|
|
||
|
|
And these are the changes required in `airflow.cfg` of our Airflow instance:
|
||
|
|
|
||
|
|
```properties
|
||
|
|
...
|
||
|
|
[openmetadata_secrets_manager]
|
||
|
|
aws_region = <aws region>
|
||
|
|
aws_access_key_id = <aws access key id>
|
||
|
|
aws_secret_access_key = <aws secret access key>
|
||
|
|
...
|
||
|
|
```
|
||
|
|
|
||
|
|
As an alternative to editing the `airflow.cfg` file, we can also set the following environment variables:
|
||
|
|
|
||
|
|
```bash
|
||
|
|
AIRFLOW__OPENMETADATA_SECRETS_MANAGER__AWS_REGION= <aws region>
|
||
|
|
AIRFLOW__OPENMETADATA_SECRETS_MANAGER__AWS_ACCESS_KEY_ID= <aws access key id>
|
||
|
|
AIRFLOW__OPENMETADATA_SECRETS_MANAGER__AWS_SECRET_ACCESS_KEY= <aws secret access key>
|
||
|
|
```
|
||
|
|
|
||
|
|
If no parameters are provided for the AWS account, or only `<aws region>`, it will use the default credentials. The
|
||
|
|
default credential will look for credentials in:
|
||
|
|
|
||
|
|
1. **Environment variables** - `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`.
|
||
|
|
2. **Shared credential file** - `~/.aws/credentials`
|
||
|
|
3. **AWS config file** - `~/.aws/config`
|
||
|
|
4. **Assume Role provider**
|
||
|
|
5. Instance metadata service on an Amazon EC2 instance that has an IAM role configured
|
||
|
|
|
||
|
|
More info in [AWS SDK for Java](https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/credentials.html) and
|
||
|
|
[Boto3 Docs](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html)
|
||
|
|
|
||
|
|
### 3. Restart both servers
|
||
|
|
|
||
|
|
After updating the configuration files, we are ready to restart both services. When the OM server starts, it will
|
||
|
|
automatically detect that a Secrets Manager has been configured and will migrate all our sensitive data and remove it
|
||
|
|
from our DB.
|
||
|
|
|
||
|
|
If everything goes as planned, all the data would be displayed using the parameters names which starts with
|
||
|
|
`/openmetadata/...` in your AWS Systems Manager Parameter Store console. The following image shows what it should look
|
||
|
|
like:
|
||
|
|
|
||
|
|
<p/>
|
||
|
|
<Image src="/images/deployment/secrets-manager/supported-implementations/aws-ssm-parameter-store/ssm-parameter-store-console.png" alt="ssm-parameter-store-console"/>
|
||
|
|
<p/>
|
||
|
|
|
||
|
|
**Note:** If we want to change the starting path for our secrets names from `openmetadata` to a different one, we have
|
||
|
|
to change the property `clusterName` in our `openmetadata.yaml`
|
||
|
|
|
||
|
|
## CLI
|
||
|
|
|
||
|
|
After enabling the Secret Manager, we also have to make a slight change in our workflows YAML files. In the
|
||
|
|
`workflowConfig` we have to add the secret manager configuration:
|
||
|
|
|
||
|
|
```yaml
|
||
|
|
workflowConfig:
|
||
|
|
openMetadataServerConfig:
|
||
|
|
secretsManagerProvider: aws-ssm
|
||
|
|
secretsManagerCredentials:
|
||
|
|
awsAccessKeyId: <aws access key id>
|
||
|
|
awsSecretAccessKey: <aws secret access key>
|
||
|
|
awsRegion: <aws region>
|
||
|
|
hostPort: <OpenMetadata host and port>
|
||
|
|
authProvider: <OpenMetadata auth provider>
|
||
|
|
```
|