mirror of
https://github.com/open-metadata/OpenMetadata.git
synced 2025-08-22 07:58:06 +00:00
Add secrets manager documentation back (#8771)
* Add secrets manager documentation back * Change OM version
This commit is contained in:
Nahuel
GitHub
parent
684cbea960
commit
2a77dd0ceb
@ -0,0 +1,90 @@
|
||||
---
|
||||
title: Secrets Manager
|
||||
slug: /deployment/secrets-manager/how-to-add-a-new-implementation
|
||||
---
|
||||
|
||||
# How to add a new implementation
|
||||
|
||||
If we want to create our implementation of a Secrets Manager, we can do it in 3 simple steps.
|
||||
|
||||
## 1. Update the JSON schema
|
||||
|
||||
Create a new entry in the JSON schema definition of the Secrets Manager provider inside the `enum` property.
|
||||
|
||||
```json
|
||||
{
|
||||
"$id": "https://open-metadata.org/schema/entity/services/connections/metadata/secretsManagerProvider.json",
|
||||
"$schema": "http://json-schema.org/draft-07/schema#",
|
||||
"title": "Secrets Manager Provider",
|
||||
"description": "OpenMetadata Secrets Manager Provider. Make sure to configure the same secrets manager providers as the ones configured on the OpenMetadata server.",
|
||||
"type": "string",
|
||||
"javaType": "org.openmetadata.schema.services.connections.metadata.SecretsManagerProvider",
|
||||
"enum": ["noop", "managed-aws","aws", "managed-aws-ssm", "aws-ssm", "in-memory", "awesome-sm"],
|
||||
"additionalProperties": false
|
||||
}
|
||||
```
|
||||
|
||||
You can find [this](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/entity/services/connections/metadata/secretsManagerProvider.json) file here in the repository.
|
||||
|
||||
## 2. Update OM Server code
|
||||
|
||||
Once we have updated the JSON Schema, we can start implementing our Secrets Manager, extending the `ExternalSecretsManager.java` abstract class located [here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-service/src/main/java/org/openmetadata/service/secrets/ThirdPartySecretsManager.java). For example:
|
||||
|
||||
```java
|
||||
public abstract class AwesomeSecretsManager extends ExternalSecretsManager {
|
||||
|
||||
protected AwesomeSecretsManager(String clusterPrefix) {
|
||||
super(SecretsManagerProvider.AWESOME_SM, clusterPrefix);
|
||||
}
|
||||
|
||||
void storeSecret(String secretName, String secretValue) {
|
||||
// your implementation
|
||||
}
|
||||
void updateSecret(String secretName, String secretValue) {
|
||||
// your implementation
|
||||
}
|
||||
|
||||
String getSecret(String secretName) {
|
||||
// your implementation
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
After this, we can update `SecretsManagerFactory.java` which is a factory class. We can find this file [here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-service/src/main/java/org/openmetadata/service/secrets/SecretsManagerFactory.java).
|
||||
|
||||
```java
|
||||
...
|
||||
case AWESOME_SM:
|
||||
return AwesomeSecretsManager.getInstance(config, clusterName);
|
||||
...
|
||||
```
|
||||
|
||||
## 3. Update Python SDK code
|
||||
|
||||
The steps are similar to the Java ones. We have to extend the [following](https://github.com/open-metadata/OpenMetadata/blob/main/ingestion/src/metadata/utils/secrets/external_secrets_manager.py) `ExternalSecretsManager` abstract class as it is shown below:
|
||||
|
||||
```python
|
||||
class AwesomeSecretsManager(ExternalSecretsManager, ABC):
|
||||
def __init__(
|
||||
self,
|
||||
cluster_prefix: str,
|
||||
):
|
||||
super().__init__(cluster_prefix, SecretsManagerProvider.awesome-sm)
|
||||
|
||||
@abstractmethod
|
||||
def get_string_value(self, name: str) -> str:
|
||||
# your implementation
|
||||
pass
|
||||
```
|
||||
|
||||
Similar to what we did in step 2, we have to add our implementation to the factory class `ExternalSecretsManager` that can be found [here]():
|
||||
|
||||
```json
|
||||
...
|
||||
elif secrets_manager_provider == SecretsManagerProvider.awesome-sm:
|
||||
return AwesomeSecretsManager(cluster_name)
|
||||
...
|
||||
```
|
||||
<p/><p/>
|
||||
|
||||
If you need support while implementing your Secret Manager client, do not hesitate to reach out to us on [Slack](https://slack.open-metadata.org/).
|
||||
@ -0,0 +1,87 @@
|
||||
---
|
||||
title: Enable Secrets Manager
|
||||
slug: /deployment/secrets-manager
|
||||
---
|
||||
|
||||
# Enable Secrets Manager
|
||||
|
||||
Secret Manager integrations allow you to use your existing third-party **Key Management Store** (KMS) with OpenMetadata.
|
||||
Your credentials and sensitive information are stored in a tool that you control, and the KMS will mediate between any
|
||||
OpenMetadata internal requirement and sensitive information.
|
||||
|
||||
Without a secret manager configured in OpenMetadata, all your sensitive data, any password field of a service connection
|
||||
parameters, bot credentials configuration or DBT configuration of an ingestion pipeline, were stored in MySQL (or
|
||||
Postgres) encrypted.
|
||||
|
||||
The following diagram shows how is the process between the OM server and Airflow workflows:
|
||||
|
||||
<p/>
|
||||
<Image src="/images/deployment/secrets-manager/om-secrets-manager-disabled.png" alt="om-secrets-manager-disabled"/>
|
||||
<p/>
|
||||
|
||||
As you can see, the `Workflow` consumed by Airflow contains the service information as an `EntityReference`. We use that
|
||||
reference to read the Service information, including its connection details. This information goes from
|
||||
`Database > OM > Airflow`.
|
||||
|
||||
When the Secrets Manager is enabled, sensitive information stop being stored in any system from OpenMetadata. Instead,
|
||||
the KMS will act as a mediator, as we can observe in the diagram below:
|
||||
|
||||
<p/>
|
||||
<Image src="/images/deployment/secrets-manager/om-secrets-manager-enabled.png" alt="om-secrets-manager-enabled"/>
|
||||
<p/>
|
||||
|
||||
In 0.13 and up, OpenMetadata will communicate through an interface to read/write sensitive information -- removing the
|
||||
need to store sensitive data in OM systems. This new interface works whether users keep using the underlying database of
|
||||
OpenMetadata to store credentials (as it was set up thus far) or any external system such as AWS Secrets Manager or AWS
|
||||
SSM Parameter Store.
|
||||
|
||||
In future releases, we will add support for additional Key Management Stores, such as Azure Key Vault or Kubernetes
|
||||
Secrets.
|
||||
|
||||
If you’d like to contribute by creating the interface, check the implementation guide, or if you want to see a new one
|
||||
on the supported list, please reach out to us on [Slack](https://slack.open-metadata.org/).
|
||||
|
||||
If you are interested in enabling the secrets' manager feature, this is our list of supported Secrets Manager
|
||||
implementations:
|
||||
|
||||
- [AWS Secrets Manager](/deployment/secrets-manager/supported-implementations/aws-secrets-manager)
|
||||
- [AWS Systems Manager Parameter Store](/deployment/secrets-manager/supported-implementations/aws-ssm-parameter-store)
|
||||
|
||||
Things to take into account when enabling the Secrets Manager feature:
|
||||
|
||||
1. The migration of all the sensitive data will be done automatically after restarting the OpenMetadata server, which
|
||||
can not be undone for the time being.
|
||||
2. Only users with permissions can edit and retrieve the service connections. The connection parameters will be hidden
|
||||
for all other users.
|
||||
|
||||
## How it works
|
||||
|
||||
There are two types of secrets manager implementations.
|
||||
|
||||
### Managed secrets manager
|
||||
|
||||
All the sensitive data will be held automatically in the configured secrets manager, i.e., any password field stored in
|
||||
the connection parameters of a service, in a bot credentials configuration, or a DBT configuration of an ingestion
|
||||
pipeline.
|
||||
|
||||
For example, suppose we create a MySQL service with the name `mysql-test`. In that case, the connection password will be
|
||||
stored in the secrets manager using the secret id `/openmetadata/database/mysql-test/password`. When we retrieve the
|
||||
connection parameters from the service, the password field will have the value
|
||||
`secrets:/openmetadata/database/mysql-test/password`.
|
||||
|
||||
We can also use secrets already stored in our secrets vault using the same convention `secret:{secret_id}`.
|
||||
|
||||
All the sensitive data (the secrets ids in this case) values will be encrypted using the Fernet algorithm as extra
|
||||
security protection.
|
||||
|
||||
### Non-managed secrets manager
|
||||
|
||||
On the other hand, the non-managed configuration allows flexibility on how we want to use our secrets vault. Instead of
|
||||
automatically storing all the sensitive data, we can use the secrets ids from our secrets vault following the convention
|
||||
`secret:{secret_id}` when filling in password fields of the connection parameters of a service, in a bot configuration,
|
||||
or a DBT configuration of an ingestion pipeline.
|
||||
|
||||
The rest of the values which don't follow the convention for using a secret will be encrypted using the Fernet algorithm
|
||||
as extra security protection.
|
||||
|
||||
|
||||
@ -0,0 +1,116 @@
|
||||
---
|
||||
title: AWS Secrets Manager
|
||||
slug: /deployment/secrets-manager/supported-implementations/aws-secrets-manager
|
||||
---
|
||||
|
||||
# AWS Secrets Manager
|
||||
|
||||
## Setup
|
||||
|
||||
The setup steps covers the use of the managed version of the AWS Secrets Manager as secrets manager but for the
|
||||
non-managed follow only the steps related to the Airflow server and CLI.
|
||||
|
||||
### 1. Permissions needed
|
||||
|
||||
These are the permissions required in the IAM policy to enable the AWS Secrets Manager in OpenMetadata.
|
||||
|
||||
```json
|
||||
{
|
||||
"Version": "2012-10-17",
|
||||
"Statement": [
|
||||
{
|
||||
"Effect": "Allow",
|
||||
"Action": [
|
||||
"secretsmanager:GetSecretValue",
|
||||
"secretsmanager:PutSecretValue",
|
||||
"secretsmanager:CreateSecret",
|
||||
"secretsmanager:UpdateSecret"
|
||||
],
|
||||
"Resource": "*"
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
### 2. Update configuration
|
||||
|
||||
We have to set up the secret manager provider we want to use, that in our case is `aws`, and the credentials for our AWS
|
||||
account.
|
||||
|
||||
The changes to be done in `openmetadata.yaml` file of the OpenMetadata server are:
|
||||
|
||||
```yaml
|
||||
...
|
||||
secretsManagerConfiguration:
|
||||
secretsManager: managed-aws # or env var SECRET_MANAGER. For non-managed use 'aws'.
|
||||
parameters:
|
||||
region: <aws region> # or env var OM_SM_REGION
|
||||
accessKeyId: <aws access key id> # or env var OM_SM_ACCESS_KEY_ID
|
||||
secretAccessKey: <aws secret access key> # or env var OM_SM_ACCESS_KEY
|
||||
...
|
||||
```
|
||||
|
||||
And these are the changes required in `airflow.cfg` of our Airflow instance:
|
||||
|
||||
```properties
|
||||
...
|
||||
[openmetadata_secrets_manager]
|
||||
aws_region = <aws region>
|
||||
aws_access_key_id = <aws access key id>
|
||||
aws_secret_access_key = <aws secret access key>
|
||||
...
|
||||
```
|
||||
|
||||
As an alternative to editing the `airflow.cfg` file, we can also set the following environment variables:
|
||||
|
||||
```bash
|
||||
AIRFLOW__OPENMETADATA_SECRETS_MANAGER__AWS_REGION= <aws region>
|
||||
AIRFLOW__OPENMETADATA_SECRETS_MANAGER__AWS_ACCESS_KEY_ID= <aws access key id>
|
||||
AIRFLOW__OPENMETADATA_SECRETS_MANAGER__AWS_SECRET_ACCESS_KEY= <aws secret access key>
|
||||
```
|
||||
|
||||
If no parameters are provided for the AWS account, or only `<aws region>`, it will use the default credentials.
|
||||
The default credential will look for credentials in:
|
||||
|
||||
1. **Environment variables** - `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`.
|
||||
2. **Shared credential file** - `~/.aws/credentials`
|
||||
3. **AWS config file** - `~/.aws/config`
|
||||
4. **Assume Role provider**
|
||||
5. Instance metadata service on an Amazon EC2 instance that has an IAM role configured
|
||||
|
||||
More info in [AWS SDK for Java](https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/credentials.html) and
|
||||
[Boto3 Docs](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html)
|
||||
|
||||
### 3. Restart both servers
|
||||
|
||||
After updating the configuration files, we are ready to restart both services. When the OM server starts, it will
|
||||
automatically detect that a Secrets Manager has been configured and will migrate all our sensitive data and remove it
|
||||
from our DB.
|
||||
|
||||
If everything goes as planned, all the data would be displayed using the secrets names which starts with
|
||||
`/openmetadata/...` in your AWS Secrets Manager console. The following image shows what it should look like:
|
||||
|
||||
<p/>
|
||||
<Image src="/images/deployment/secrets-manager/supported-implementations/aws-secrets-manager/secrets-manager-console.png" alt="secrets-manager-console"/>
|
||||
<p/>
|
||||
|
||||
**Note:** If we want to change the starting path for our secrets names from `openmetadata` to a different one, we have
|
||||
to change the property `clusterName` in our `openmetadata.yaml`
|
||||
|
||||
## CLI
|
||||
|
||||
After enabling the Secret Manager, we also have to make a slight change in our workflows YAML files. In the
|
||||
`workflowConfig` we have to add the secret manager configuration:
|
||||
|
||||
```yaml
|
||||
workflowConfig:
|
||||
openMetadataServerConfig:
|
||||
secretsManagerProvider: aws
|
||||
secretsManagerCredentials:
|
||||
awsAccessKeyId: <aws access key id>
|
||||
awsSecretAccessKey: <aws secret access key>
|
||||
awsRegion: <aws region>
|
||||
hostPort: <OpenMetadata host and port>
|
||||
authProvider: <OpenMetadata auth provider>
|
||||
```
|
||||
|
||||
@ -0,0 +1,114 @@
|
||||
---
|
||||
title: AWS Systems Manager Parameter Store
|
||||
slug: /deployment/secrets-manager/supported-implementations/aws-ssm-parameter-store
|
||||
---
|
||||
|
||||
# AWS Systems Manager Parameter Store
|
||||
|
||||
The setup steps covers the use of the managed version of the AWS Systems Manager Parameter Store as secrets manager but
|
||||
for the non-managed follow only the steps related to the Airflow server and CLI.
|
||||
|
||||
## Setup
|
||||
|
||||
### 1. Permissions needed
|
||||
|
||||
These are the permissions required in the IAM policy to enable the AWS Systems Manager Parameter Store in OpenMetadata.
|
||||
|
||||
```json
|
||||
{
|
||||
"Version": "2012-10-17",
|
||||
"Statement": [
|
||||
{
|
||||
"Effect": "Allow",
|
||||
"Action": [
|
||||
"ssm:PutParameter",
|
||||
"ssm:GetParameter"
|
||||
],
|
||||
"Resource": "*"
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
### 2. Update configuration
|
||||
|
||||
We have to set up the secret manager provider we want to use, that in our case is `aws-ssm`, and the credentials for our
|
||||
AWS account.
|
||||
|
||||
The changes to be done in `openmetadata.yaml` file of the OpenMetadata server are:
|
||||
|
||||
```yaml
|
||||
...
|
||||
secretsManagerConfiguration:
|
||||
secretsManager: managed-aws-ssm # or env var SECRET_MANAGER. For non-managed use 'aws-ssm'.
|
||||
parameters:
|
||||
region: <aws region> # or env var OM_SM_REGION
|
||||
accessKeyId: <aws access key id> # or env var OM_SM_ACCESS_KEY_ID
|
||||
secretAccessKey: <aws secret access key> # or env var OM_SM_ACCESS_KEY
|
||||
...
|
||||
```
|
||||
|
||||
And these are the changes required in `airflow.cfg` of our Airflow instance:
|
||||
|
||||
```properties
|
||||
...
|
||||
[openmetadata_secrets_manager]
|
||||
aws_region = <aws region>
|
||||
aws_access_key_id = <aws access key id>
|
||||
aws_secret_access_key = <aws secret access key>
|
||||
...
|
||||
```
|
||||
|
||||
As an alternative to editing the `airflow.cfg` file, we can also set the following environment variables:
|
||||
|
||||
```bash
|
||||
AIRFLOW__OPENMETADATA_SECRETS_MANAGER__AWS_REGION= <aws region>
|
||||
AIRFLOW__OPENMETADATA_SECRETS_MANAGER__AWS_ACCESS_KEY_ID= <aws access key id>
|
||||
AIRFLOW__OPENMETADATA_SECRETS_MANAGER__AWS_SECRET_ACCESS_KEY= <aws secret access key>
|
||||
```
|
||||
|
||||
If no parameters are provided for the AWS account, or only `<aws region>`, it will use the default credentials. The
|
||||
default credential will look for credentials in:
|
||||
|
||||
1. **Environment variables** - `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`.
|
||||
2. **Shared credential file** - `~/.aws/credentials`
|
||||
3. **AWS config file** - `~/.aws/config`
|
||||
4. **Assume Role provider**
|
||||
5. Instance metadata service on an Amazon EC2 instance that has an IAM role configured
|
||||
|
||||
More info in [AWS SDK for Java](https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/credentials.html) and
|
||||
[Boto3 Docs](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html)
|
||||
|
||||
### 3. Restart both servers
|
||||
|
||||
After updating the configuration files, we are ready to restart both services. When the OM server starts, it will
|
||||
automatically detect that a Secrets Manager has been configured and will migrate all our sensitive data and remove it
|
||||
from our DB.
|
||||
|
||||
If everything goes as planned, all the data would be displayed using the parameters names which starts with
|
||||
`/openmetadata/...` in your AWS Systems Manager Parameter Store console. The following image shows what it should look
|
||||
like:
|
||||
|
||||
<p/>
|
||||
<Image src="/images/deployment/secrets-manager/supported-implementations/aws-ssm-parameter-store/ssm-parameter-store-console.png" alt="ssm-parameter-store-console"/>
|
||||
<p/>
|
||||
|
||||
**Note:** If we want to change the starting path for our secrets names from `openmetadata` to a different one, we have
|
||||
to change the property `clusterName` in our `openmetadata.yaml`
|
||||
|
||||
## CLI
|
||||
|
||||
After enabling the Secret Manager, we also have to make a slight change in our workflows YAML files. In the
|
||||
`workflowConfig` we have to add the secret manager configuration:
|
||||
|
||||
```yaml
|
||||
workflowConfig:
|
||||
openMetadataServerConfig:
|
||||
secretsManagerProvider: aws-ssm
|
||||
secretsManagerCredentials:
|
||||
awsAccessKeyId: <aws access key id>
|
||||
awsSecretAccessKey: <aws secret access key>
|
||||
awsRegion: <aws region>
|
||||
hostPort: <OpenMetadata host and port>
|
||||
authProvider: <OpenMetadata auth provider>
|
||||
```
|
||||
@ -0,0 +1,27 @@
|
||||
---
|
||||
title: Secrets Manager
|
||||
slug: /deployment/secrets-manager/supported-implementations
|
||||
---
|
||||
|
||||
# Supported implementations
|
||||
|
||||
This is our list of supported Secrets Manager implementations:
|
||||
|
||||
<InlineCalloutContainer>
|
||||
<InlineCallout
|
||||
color="violet-70"
|
||||
bold="AWS Secrets Manager"
|
||||
icon="vpn_key"
|
||||
href="/deployment/secrets-manager/supported-implementations/aws-secrets-manager"
|
||||
>
|
||||
AWS Secrets Manager
|
||||
</InlineCallout>
|
||||
<InlineCallout
|
||||
color="violet-70"
|
||||
bold="AWS Systems Manager Parameter Store"
|
||||
icon="vpn_key"
|
||||
href="/deployment/secrets-manager/supported-implementations/aws-ssm-parameter-store"
|
||||
>
|
||||
AWS Systems Manager Parameter Store
|
||||
</InlineCallout>
|
||||
</InlineCalloutContainer>
|
||||
@ -134,6 +134,17 @@ site_menu:
|
||||
- category: Deployment / Enable Security / Enable JWT Tokens
|
||||
url: /deployment/security/enable-jwt-tokens
|
||||
|
||||
- category: Deployment / Enable Secrets Manager
|
||||
url: /deployment/secrets-manager
|
||||
- category: Deployment / Enable Secrets Manager / Supported Implementations
|
||||
url: /deployment/secrets-manager/supported-implementations
|
||||
- category: Deployment / Enable Secrets Manager / Supported Implementations / AWS Secrets Manager
|
||||
url: /deployment/secrets-manager/supported-implementations/aws-secrets-manager
|
||||
- category: Deployment / Enable Secrets Manager / Supported Implementations / AWS SSM Parameter Store
|
||||
url: /deployment/secrets-manager/supported-implementations/aws-ssm-parameter-store
|
||||
- category: Deployment / Enable Secrets Manager / How to add a new implementation
|
||||
url: /deployment/secrets-manager/how-to-add-a-new-implementation
|
||||
|
||||
- category: Deployment / Upgrade OpenMetadata
|
||||
url: /deployment/upgrade
|
||||
- category: Deployment / Upgrade OpenMetadata / Upgrade on Bare Metal
|
||||
|
||||
@ -1,5 +1,5 @@
|
||||
{
|
||||
"$id": "https://open-metadata.org/schema/entity/services/connections/metadata/secretsManagersProvider.json",
|
||||
"$id": "https://open-metadata.org/schema/entity/services/connections/metadata/secretsManagerProvider.json",
|
||||
"$schema": "http://json-schema.org/draft-07/schema#",
|
||||
"title": "Secrets Manager Provider",
|
||||
"description": "OpenMetadata Secrets Manager Provider. Make sure to configure the same secrets manager providers as the ones configured on the OpenMetadata server.",
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user