mirror of
https://github.com/open-metadata/OpenMetadata.git
synced 2025-10-18 12:18:35 +00:00
Temporary remove of secrets manager docs (#7917)
This commit is contained in:
Nahuel
GitHub
parent
f3e6266b7a
commit
3c99184e52
@ -1,90 +0,0 @@
|
||||
---
|
||||
title: Secrets Manager
|
||||
slug: /deployment/secrets-manager/how-to-add-a-new-implementation
|
||||
---
|
||||
|
||||
# How to add a new implementation
|
||||
|
||||
If we want to create our implementation of a Secrets Manager, we can do it in 3 simple steps.
|
||||
|
||||
## 1. Update the JSON schema
|
||||
|
||||
Create a new entry in the JSON schema definition of the Secrets Manager provider inside the `enum` property.
|
||||
|
||||
```json
|
||||
{
|
||||
"$id": "https://open-metadata.org/schema/entity/services/connections/metadata/secretsManagersProvider.json",
|
||||
"$schema": "http://json-schema.org/draft-07/schema#",
|
||||
"title": "Secrets Manager Provider",
|
||||
"description": "OpenMetadata Secrets Manager Provider. Make sure to configure the same secrets manager providers as the ones configured on the OpenMetadata server.",
|
||||
"type": "string",
|
||||
"javaType": "org.openmetadata.catalog.services.connections.metadata.SecretsManagerProvider",
|
||||
"enum": ["noop", "aws", "aws-ssm", "awesome-sm"],
|
||||
"additionalProperties": false
|
||||
}
|
||||
```
|
||||
|
||||
You can find [this](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/entity/services/connections/metadata/secretsManagerProvider.json) file here in the repository.
|
||||
|
||||
## 2. Update OM Server code
|
||||
|
||||
Once we have updated the JSON Schema, we can start implementing our Secrets Manager, extending the `ThirdPartySecretsManager.java` abstract class located [here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-service/src/main/java/org/openmetadata/service/secrets/ThirdPartySecretsManager.java). For example:
|
||||
|
||||
```java
|
||||
public abstract class AwesomeSecretsManager extends ThirdPartySecretsManager {
|
||||
|
||||
protected AwesomeSecretsManager(String clusterPrefix) {
|
||||
super(SecretsManagerProvider.AWESOME_SM, clusterPrefix);
|
||||
}
|
||||
|
||||
void storeSecret(String secretName, String secretValue) {
|
||||
// your implementation
|
||||
}
|
||||
void updateSecret(String secretName, String secretValue) {
|
||||
// your implementation
|
||||
}
|
||||
|
||||
String getSecret(String secretName) {
|
||||
// your implementation
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
After this, we can update `SecretsManagerFactory.java` which is a factory class. We can find this file [here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-service/src/main/java/org/openmetadata/service/secrets/SecretsManagerFactory.java).
|
||||
|
||||
```java
|
||||
...
|
||||
case AWESOME_SM:
|
||||
return AwesomeSecretsManager.getInstance(config, clusterName);
|
||||
...
|
||||
```
|
||||
|
||||
## 3. Update Python SDK code
|
||||
|
||||
The steps are similar to the Java ones. We have to extend the [following](https://github.com/open-metadata/OpenMetadata/blob/main/ingestion/src/metadata/utils/secrets/external_secrets_manager.py) `ExternalSecretsManager` abstract class as it is shown below:
|
||||
|
||||
```python
|
||||
class AwesomeSecretsManager(ExternalSecretsManager, ABC):
|
||||
def __init__(
|
||||
self,
|
||||
cluster_prefix: str,
|
||||
):
|
||||
super().__init__(cluster_prefix, SecretsManagerProvider.awesome-sm)
|
||||
|
||||
@abstractmethod
|
||||
def get_string_value(self, name: str) -> str:
|
||||
# your implementation
|
||||
pass
|
||||
```
|
||||
|
||||
Similar to what we did in step 2, we have to add our implementation to the factory class `ExternalSecretsManager` that can be found [here]():
|
||||
|
||||
```json
|
||||
...
|
||||
elif secrets_manager_provider == SecretsManagerProvider.awesome-sm:
|
||||
return AwesomeSecretsManager(cluster_name)
|
||||
...
|
||||
```
|
||||
<p/><p/>
|
||||
|
||||
If you need support while implementing your Secret Manager client, do not hesitate to reach out to us on [Slack](https://slack.open-metadata.org/).
|
||||
@ -1,40 +0,0 @@
|
||||
---
|
||||
title: Enable Secrets Manager
|
||||
slug: /deployment/secrets-manager
|
||||
---
|
||||
|
||||
# Enable Secrets Manager
|
||||
|
||||
Secret Manager integrations allow you to use your existing third-party **Key Management Store** (KMS) with OpenMetadata. Your credentials and sensitive information are stored in a tool that you control, and the KMS will mediate between any OpenMetadata internal requirement and sensitive information.
|
||||
|
||||
Without a secret manager configured in OpenMetadata, all your sensitive data, such as service connection or auth provider configuration, were stored in MySQL (or Postgres) encrypted.
|
||||
|
||||
The following diagram shows how is the process between the OM server and Airflow workflows:
|
||||
|
||||
<p/>
|
||||
<Image src="/images/deployment/secrets-manager/om-secrets-manager-disabled.png" alt="om-secrets-manager-disabled"/>
|
||||
<p/>
|
||||
|
||||
As you can see, the `Workflow` consumed by Airflow contains the service information as an `EntityReference`. We use that reference to read the Service information, including its connection details. This information goes from `Database > OM > Airflow`.
|
||||
|
||||
When the Secrets Manager is enabled, sensitive information stop being stored in any system from OpenMetadata. Instead, the KMS will act as a mediator, as we can observe in the diagram below:
|
||||
|
||||
<p/>
|
||||
<Image src="/images/deployment/secrets-manager/om-secrets-manager-enabled.png" alt="om-secrets-manager-enabled"/>
|
||||
<p/>
|
||||
|
||||
In 0.12 and up, OpenMetadata will communicate through an interface to read/write sensitive information -- removing the need to store sensitive data in OM systems. This new interface works whether users keep using the underlying database of OpenMetadata to store credentials (as it was setup thus far) or any external system such as AWS Secrets Manager or AWS SSM Parameter Store.
|
||||
|
||||
In future releases, we will add support for additional Key Management Stores, such as Azure Key Vault or Kubernetes Secrets.
|
||||
|
||||
If you’d like to contribute by creating the interface, check the implementation guide, or if you want to see a new one on the supported list, please reach out to us on [Slack](https://slack.open-metadata.org/).
|
||||
|
||||
If you are interested in enabling the secrets manager feature, this is our list of supported Secrets Manager implementations:
|
||||
|
||||
- [AWS Secrets Manager](/deployment/secrets-manager/supported-implementations/aws-secrets-manager)
|
||||
- [AWS Systems Manager Parameter Store](/deployment/secrets-manager/supported-implementations/aws-ssm-parameter-store)
|
||||
|
||||
Things to take into account when enabling the Secrets Manager feature:
|
||||
|
||||
1. The migration of all the sensitive data will be done automatically after restarting the OpenMetadata server, which can not be undone for the time being.
|
||||
2. Only admin users can retrieve or edit the service connections. The connection parameters will be hidden for all other users including bots.
|
||||
@ -1,105 +0,0 @@
|
||||
---
|
||||
title: AWS Secrets Manager
|
||||
slug: /deployment/secrets-manager/supported-implementations/aws-secrets-manager
|
||||
---
|
||||
|
||||
# AWS Secrets Manager
|
||||
|
||||
## Setup
|
||||
|
||||
### 1. Permissions needed
|
||||
|
||||
These are the permissions required in the IAM policy to enable the AWS Secrets Manager in OpenMetadata.
|
||||
|
||||
```json
|
||||
{
|
||||
"Version": "2012-10-17",
|
||||
"Statement": [
|
||||
{
|
||||
"Effect": "Allow",
|
||||
"Action": [
|
||||
"secretsmanager:GetSecretValue",
|
||||
"secretsmanager:PutSecretValue",
|
||||
"secretsmanager:CreateSecret",
|
||||
"secretsmanager:UpdateSecret"
|
||||
],
|
||||
"Resource": "*"
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
### 2. Update configuration
|
||||
|
||||
We have to set up the secret manager provider we want to use, that in our case is `aws`, and the credentials for our AWS account.
|
||||
|
||||
The changes to be done in `openmetadata.yaml` file of the OpenMetadata server are:
|
||||
|
||||
```yaml
|
||||
...
|
||||
secretsManagerConfiguration:
|
||||
secretsManager: aws # or env var SECRET_MANAGER
|
||||
parameters:
|
||||
region: <aws region> # or env var OM_SM_REGION
|
||||
accessKeyId: <aws access key id> # or env var OM_SM_ACCESS_KEY_ID
|
||||
secretAccessKey: <aws secret access key> # or env var OM_SM_ACCESS_KEY
|
||||
...
|
||||
```
|
||||
|
||||
And these are the changes required in `airflow.cfg` of our Airflow instance:
|
||||
|
||||
```properties
|
||||
...
|
||||
[openmetadata_secrets_manager]
|
||||
aws_region = <aws region>
|
||||
aws_access_key_id = <aws access key id>
|
||||
aws_secret_access_key = <aws secret access key>
|
||||
...
|
||||
```
|
||||
|
||||
As an alternative to editing the `airflow.cfg` file, we can also set the following environment variables:
|
||||
|
||||
```bash
|
||||
AIRFLOW__OPENMETADATA_SECRETS_MANAGER__AWS_REGION= <aws region>
|
||||
AIRFLOW__OPENMETADATA_SECRETS_MANAGER__AWS_ACCESS_KEY_ID= <aws access key id>
|
||||
AIRFLOW__OPENMETADATA_SECRETS_MANAGER__AWS_SECRET_ACCESS_KEY= <aws secret access key>
|
||||
```
|
||||
|
||||
If no parameters are provided for the AWS account, or only `<aws region>`, it will use the default credentials. The default credential will look for credentials in:
|
||||
|
||||
1. **Environment variables** - `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`.
|
||||
2. **Shared credential file** - `~/.aws/credentials`
|
||||
3. **AWS config file** - `~/.aws/config`
|
||||
4. **Assume Role provider**
|
||||
5. Instance metadata service on an Amazon EC2 instance that has an IAM role configured
|
||||
|
||||
More info in [AWS SDK for Java](https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/credentials.html) and [Boto3 Docs](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html)
|
||||
|
||||
### 3. Restart both servers
|
||||
|
||||
After updating the configuration files, we are ready to restart both services. When the OM server starts, it will automatically detect that a Secrets Manager has been configured and will migrate all our sensitive data and remove it from our DB.
|
||||
|
||||
If everything goes as planned, all the data would be displayed using the secrets names which starts with `/openmetadata/...` in your AWS Secrets Manager console. The following image shows what it should look like:
|
||||
|
||||
<p/>
|
||||
<Image src="/images/deployment/secrets-manager/supported-implementations/aws-secrets-manager/secrets-manager-console.png" alt="secrets-manager-console"/>
|
||||
<p/>
|
||||
|
||||
**Note:** If we want to change the starting path for our secrets names from `openmetadata` to a different one, we have to change the property `clusterName` in our `openmetadata.yaml`
|
||||
|
||||
## CLI
|
||||
|
||||
After enabling the Secret Manager, we also have to make a slight change in our workflows YAML files. In the `workflowConfig` we have to add the secret manager configuration:
|
||||
|
||||
```yaml
|
||||
workflowConfig:
|
||||
openMetadataServerConfig:
|
||||
secretsManagerProvider: aws
|
||||
secretsManagerCredentials:
|
||||
awsAccessKeyId: <aws access key id>
|
||||
awsSecretAccessKey: <aws secret access key>
|
||||
awsRegion: <aws region>
|
||||
hostPort: <OpenMetadata host and port>
|
||||
authProvider: <OpenMetadata auth provider>
|
||||
```
|
||||
|
||||
@ -1,102 +0,0 @@
|
||||
---
|
||||
title: AWS Systems Manager Parameter Store
|
||||
slug: /deployment/secrets-manager/supported-implementations/aws-ssm-parameter-store
|
||||
---
|
||||
|
||||
# AWS Systems Manager Parameter Store
|
||||
|
||||
## Setup
|
||||
|
||||
### 1. Permissions needed
|
||||
|
||||
These are the permissions required in the IAM policy to enable the AWS Systems Manager Parameter Store in OpenMetadata.
|
||||
|
||||
```json
|
||||
{
|
||||
"Version": "2012-10-17",
|
||||
"Statement": [
|
||||
{
|
||||
"Effect": "Allow",
|
||||
"Action": [
|
||||
"ssm:PutParameter",
|
||||
"ssm:GetParameter"
|
||||
],
|
||||
"Resource": "*"
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
### 2. Update configuration
|
||||
|
||||
We have to set up the secret manager provider we want to use, that in our case is `aws-ssm`, and the credentials for our AWS account.
|
||||
|
||||
The changes to be done in `openmetadata.yaml` file of the OpenMetadata server are:
|
||||
|
||||
```yaml
|
||||
...
|
||||
secretsManagerConfiguration:
|
||||
secretsManager: aws-ssm # or env var SECRET_MANAGER
|
||||
parameters:
|
||||
region: <aws region> # or env var OM_SM_REGION
|
||||
accessKeyId: <aws access key id> # or env var OM_SM_ACCESS_KEY_ID
|
||||
secretAccessKey: <aws secret access key> # or env var OM_SM_ACCESS_KEY
|
||||
...
|
||||
```
|
||||
|
||||
And these are the changes required in `airflow.cfg` of our Airflow instance:
|
||||
|
||||
```properties
|
||||
...
|
||||
[openmetadata_secrets_manager]
|
||||
aws_region = <aws region>
|
||||
aws_access_key_id = <aws access key id>
|
||||
aws_secret_access_key = <aws secret access key>
|
||||
...
|
||||
```
|
||||
|
||||
As an alternative to editing the `airflow.cfg` file, we can also set the following environment variables:
|
||||
|
||||
```bash
|
||||
AIRFLOW__OPENMETADATA_SECRETS_MANAGER__AWS_REGION= <aws region>
|
||||
AIRFLOW__OPENMETADATA_SECRETS_MANAGER__AWS_ACCESS_KEY_ID= <aws access key id>
|
||||
AIRFLOW__OPENMETADATA_SECRETS_MANAGER__AWS_SECRET_ACCESS_KEY= <aws secret access key>
|
||||
```
|
||||
|
||||
If no parameters are provided for the AWS account, or only `<aws region>`, it will use the default credentials. The default credential will look for credentials in:
|
||||
|
||||
1. **Environment variables** - `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`.
|
||||
2. **Shared credential file** - `~/.aws/credentials`
|
||||
3. **AWS config file** - `~/.aws/config`
|
||||
4. **Assume Role provider**
|
||||
5. Instance metadata service on an Amazon EC2 instance that has an IAM role configured
|
||||
|
||||
More info in [AWS SDK for Java](https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/credentials.html) and [Boto3 Docs](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html)
|
||||
|
||||
### 3. Restart both servers
|
||||
|
||||
After updating the configuration files, we are ready to restart both services. When the OM server starts, it will automatically detect that a Secrets Manager has been configured and will migrate all our sensitive data and remove it from our DB.
|
||||
|
||||
If everything goes as planned, all the data would be displayed using the parameters names which starts with `/openmetadata/...` in your AWS Systems Manager Parameter Store console. The following image shows what it should look like:
|
||||
|
||||
<p/>
|
||||
<Image src="/images/deployment/secrets-manager/supported-implementations/aws-ssm-parameter-store/ssm-parameter-store-console.png" alt="ssm-parameter-store-console"/>
|
||||
<p/>
|
||||
|
||||
**Note:** If we want to change the starting path for our secrets names from `openmetadata` to a different one, we have to change the property `clusterName` in our `openmetadata.yaml`
|
||||
|
||||
## CLI
|
||||
|
||||
After enabling the Secret Manager, we also have to make a slight change in our workflows YAML files. In the `workflowConfig` we have to add the secret manager configuration:
|
||||
|
||||
```yaml
|
||||
workflowConfig:
|
||||
openMetadataServerConfig:
|
||||
secretsManagerProvider: aws-ssm
|
||||
secretsManagerCredentials:
|
||||
awsAccessKeyId: <aws access key id>
|
||||
awsSecretAccessKey: <aws secret access key>
|
||||
awsRegion: <aws region>
|
||||
hostPort: <OpenMetadata host and port>
|
||||
authProvider: <OpenMetadata auth provider>
|
||||
```
|
||||
@ -1,27 +0,0 @@
|
||||
---
|
||||
title: Secrets Manager
|
||||
slug: /deployment/secrets-manager/supported-implementations
|
||||
---
|
||||
|
||||
# Supported implementations
|
||||
|
||||
This is our list of supported Secrets Manager implementations:
|
||||
|
||||
<InlineCalloutContainer>
|
||||
<InlineCallout
|
||||
color="violet-70"
|
||||
bold="AWS Secrets Manager"
|
||||
icon="vpn_key"
|
||||
href="/deployment/secrets-manager/supported-implementations/aws-secrets-manager"
|
||||
>
|
||||
AWS Secrets Manager
|
||||
</InlineCallout>
|
||||
<InlineCallout
|
||||
color="violet-70"
|
||||
bold="AWS Systems Manager Parameter Store"
|
||||
icon="vpn_key"
|
||||
href="/deployment/secrets-manager/supported-implementations/aws-ssm-parameter-store"
|
||||
>
|
||||
AWS Systems Manager Parameter Store
|
||||
</InlineCallout>
|
||||
</InlineCalloutContainer>
|
||||
@ -132,17 +132,6 @@ site_menu:
|
||||
- category: Deployment / Enable Security / Enable JWT Tokens
|
||||
url: /deployment/security/enable-jwt-tokens
|
||||
|
||||
- category: Deployment / Enable Secrets Manager
|
||||
url: /deployment/secrets-manager
|
||||
- category: Deployment / Enable Secrets Manager / Supported Implementations
|
||||
url: /deployment/secrets-manager/supported-implementations
|
||||
- category: Deployment / Enable Secrets Manager / Supported Implementations / AWS Secrets Manager
|
||||
url: /deployment/secrets-manager/supported-implementations/aws-secrets-manager
|
||||
- category: Deployment / Enable Secrets Manager / Supported Implementations / AWS SSM Parameter Store
|
||||
url: /deployment/secrets-manager/supported-implementations/aws-ssm-parameter-store
|
||||
- category: Deployment / Enable Secrets Manager / How to add a new implementation
|
||||
url: /deployment/secrets-manager/how-to-add-a-new-implementation
|
||||
|
||||
- category: Deployment / Upgrade OpenMetadata
|
||||
url: /deployment/upgrade
|
||||
- category: Deployment / Upgrade OpenMetadata / Backup Metadata
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user