diff --git a/openmetadata-docs/content/deployment/secrets-manager/how-to-add-a-new-implementation/index.md b/openmetadata-docs/content/deployment/secrets-manager/how-to-add-a-new-implementation/index.md new file mode 100644 index 00000000000..ff913322c5e --- /dev/null +++ b/openmetadata-docs/content/deployment/secrets-manager/how-to-add-a-new-implementation/index.md @@ -0,0 +1,90 @@ +--- +title: Secrets Manager +slug: /deployment/secrets-manager/how-to-add-a-new-implementation +--- + +# How to add a new implementation + +If we want to create our implementation of a Secrets Manager, we can do it in 3 simple steps. + +## 1. Update the JSON schema + +Create a new entry in the JSON schema definition of the Secrets Manager provider inside the `enum` property. + +```json +{ + "$id": "https://open-metadata.org/schema/entity/services/connections/metadata/secretsManagerProvider.json", + "$schema": "http://json-schema.org/draft-07/schema#", + "title": "Secrets Manager Provider", + "description": "OpenMetadata Secrets Manager Provider. Make sure to configure the same secrets manager providers as the ones configured on the OpenMetadata server.", + "type": "string", + "javaType": "org.openmetadata.schema.services.connections.metadata.SecretsManagerProvider", + "enum": ["noop", "managed-aws","aws", "managed-aws-ssm", "aws-ssm", "in-memory", "awesome-sm"], + "additionalProperties": false +} +``` + +You can find [this](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/entity/services/connections/metadata/secretsManagerProvider.json) file here in the repository. + +## 2. Update OM Server code + +Once we have updated the JSON Schema, we can start implementing our Secrets Manager, extending the `ExternalSecretsManager.java` abstract class located [here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-service/src/main/java/org/openmetadata/service/secrets/ThirdPartySecretsManager.java). For example: + +```java +public abstract class AwesomeSecretsManager extends ExternalSecretsManager { + + protected AwesomeSecretsManager(String clusterPrefix) { + super(SecretsManagerProvider.AWESOME_SM, clusterPrefix); + } + + void storeSecret(String secretName, String secretValue) { + // your implementation + } + void updateSecret(String secretName, String secretValue) { + // your implementation + } + + String getSecret(String secretName) { + // your implementation + } +} +``` + +After this, we can update `SecretsManagerFactory.java` which is a factory class. We can find this file [here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-service/src/main/java/org/openmetadata/service/secrets/SecretsManagerFactory.java). + +```java +... + case AWESOME_SM: + return AwesomeSecretsManager.getInstance(config, clusterName); +... +``` + +## 3. Update Python SDK code + +The steps are similar to the Java ones. We have to extend the [following](https://github.com/open-metadata/OpenMetadata/blob/main/ingestion/src/metadata/utils/secrets/external_secrets_manager.py) `ExternalSecretsManager` abstract class as it is shown below: + +```python +class AwesomeSecretsManager(ExternalSecretsManager, ABC): + def __init__( + self, + cluster_prefix: str, + ): + super().__init__(cluster_prefix, SecretsManagerProvider.awesome-sm) + + @abstractmethod + def get_string_value(self, name: str) -> str: + # your implementation + pass +``` + +Similar to what we did in step 2, we have to add our implementation to the factory class `ExternalSecretsManager` that can be found [here](): + +```json +... + elif secrets_manager_provider == SecretsManagerProvider.awesome-sm: + return AwesomeSecretsManager(cluster_name) +... +``` +

+ +If you need support while implementing your Secret Manager client, do not hesitate to reach out to us on [Slack](https://slack.open-metadata.org/). diff --git a/openmetadata-docs/content/deployment/secrets-manager/index.md b/openmetadata-docs/content/deployment/secrets-manager/index.md new file mode 100644 index 00000000000..bee17f8420a --- /dev/null +++ b/openmetadata-docs/content/deployment/secrets-manager/index.md @@ -0,0 +1,87 @@ +--- +title: Enable Secrets Manager +slug: /deployment/secrets-manager +--- + +# Enable Secrets Manager + +Secret Manager integrations allow you to use your existing third-party **Key Management Store** (KMS) with OpenMetadata. +Your credentials and sensitive information are stored in a tool that you control, and the KMS will mediate between any +OpenMetadata internal requirement and sensitive information. + +Without a secret manager configured in OpenMetadata, all your sensitive data, any password field of a service connection +parameters, bot credentials configuration or DBT configuration of an ingestion pipeline, were stored in MySQL (or +Postgres) encrypted. + +The following diagram shows how is the process between the OM server and Airflow workflows: + +

+om-secrets-manager-disabled +

+ +As you can see, the `Workflow` consumed by Airflow contains the service information as an `EntityReference`. We use that +reference to read the Service information, including its connection details. This information goes from +`Database > OM > Airflow`. + +When the Secrets Manager is enabled, sensitive information stop being stored in any system from OpenMetadata. Instead, +the KMS will act as a mediator, as we can observe in the diagram below: + +

+om-secrets-manager-enabled +

+ +In 0.13 and up, OpenMetadata will communicate through an interface to read/write sensitive information -- removing the +need to store sensitive data in OM systems. This new interface works whether users keep using the underlying database of +OpenMetadata to store credentials (as it was set up thus far) or any external system such as AWS Secrets Manager or AWS +SSM Parameter Store. + +In future releases, we will add support for additional Key Management Stores, such as Azure Key Vault or Kubernetes +Secrets. + +If you’d like to contribute by creating the interface, check the implementation guide, or if you want to see a new one +on the supported list, please reach out to us on [Slack](https://slack.open-metadata.org/). + +If you are interested in enabling the secrets' manager feature, this is our list of supported Secrets Manager +implementations: + +- [AWS Secrets Manager](/deployment/secrets-manager/supported-implementations/aws-secrets-manager) +- [AWS Systems Manager Parameter Store](/deployment/secrets-manager/supported-implementations/aws-ssm-parameter-store) + +Things to take into account when enabling the Secrets Manager feature: + +1. The migration of all the sensitive data will be done automatically after restarting the OpenMetadata server, which +can not be undone for the time being. +2. Only users with permissions can edit and retrieve the service connections. The connection parameters will be hidden +for all other users. + +## How it works + +There are two types of secrets manager implementations. + +### Managed secrets manager + +All the sensitive data will be held automatically in the configured secrets manager, i.e., any password field stored in +the connection parameters of a service, in a bot credentials configuration, or a DBT configuration of an ingestion +pipeline. + +For example, suppose we create a MySQL service with the name `mysql-test`. In that case, the connection password will be +stored in the secrets manager using the secret id `/openmetadata/database/mysql-test/password`. When we retrieve the +connection parameters from the service, the password field will have the value +`secrets:/openmetadata/database/mysql-test/password`. + +We can also use secrets already stored in our secrets vault using the same convention `secret:{secret_id}`. + +All the sensitive data (the secrets ids in this case) values will be encrypted using the Fernet algorithm as extra +security protection. + +### Non-managed secrets manager + +On the other hand, the non-managed configuration allows flexibility on how we want to use our secrets vault. Instead of +automatically storing all the sensitive data, we can use the secrets ids from our secrets vault following the convention +`secret:{secret_id}` when filling in password fields of the connection parameters of a service, in a bot configuration, +or a DBT configuration of an ingestion pipeline. + +The rest of the values which don't follow the convention for using a secret will be encrypted using the Fernet algorithm +as extra security protection. + + diff --git a/openmetadata-docs/content/deployment/secrets-manager/supported-implementations/aws-secrets-manager/index.md b/openmetadata-docs/content/deployment/secrets-manager/supported-implementations/aws-secrets-manager/index.md new file mode 100644 index 00000000000..3dbd3a797d7 --- /dev/null +++ b/openmetadata-docs/content/deployment/secrets-manager/supported-implementations/aws-secrets-manager/index.md @@ -0,0 +1,116 @@ +--- +title: AWS Secrets Manager +slug: /deployment/secrets-manager/supported-implementations/aws-secrets-manager +--- + +# AWS Secrets Manager + +## Setup + +The setup steps covers the use of the managed version of the AWS Secrets Manager as secrets manager but for the +non-managed follow only the steps related to the Airflow server and CLI. + +### 1. Permissions needed + +These are the permissions required in the IAM policy to enable the AWS Secrets Manager in OpenMetadata. + +```json +{ + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Action": [ + "secretsmanager:GetSecretValue", + "secretsmanager:PutSecretValue", + "secretsmanager:CreateSecret", + "secretsmanager:UpdateSecret" + ], + "Resource": "*" + } + ] +} +``` + +### 2. Update configuration + +We have to set up the secret manager provider we want to use, that in our case is `aws`, and the credentials for our AWS +account. + +The changes to be done in `openmetadata.yaml` file of the OpenMetadata server are: + +```yaml +... +secretsManagerConfiguration: + secretsManager: managed-aws # or env var SECRET_MANAGER. For non-managed use 'aws'. + parameters: + region: # or env var OM_SM_REGION + accessKeyId: # or env var OM_SM_ACCESS_KEY_ID + secretAccessKey: # or env var OM_SM_ACCESS_KEY +... +``` + +And these are the changes required in `airflow.cfg` of our Airflow instance: + +```properties +... +[openmetadata_secrets_manager] +aws_region = +aws_access_key_id = +aws_secret_access_key = +... +``` + +As an alternative to editing the `airflow.cfg` file, we can also set the following environment variables: + +```bash +AIRFLOW__OPENMETADATA_SECRETS_MANAGER__AWS_REGION= +AIRFLOW__OPENMETADATA_SECRETS_MANAGER__AWS_ACCESS_KEY_ID= +AIRFLOW__OPENMETADATA_SECRETS_MANAGER__AWS_SECRET_ACCESS_KEY= +``` + +If no parameters are provided for the AWS account, or only ``, it will use the default credentials. +The default credential will look for credentials in: + +1. **Environment variables** - `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`. +2. **Shared credential file** - `~/.aws/credentials` +3. **AWS config file** - `~/.aws/config` +4. **Assume Role provider** +5. Instance metadata service on an Amazon EC2 instance that has an IAM role configured + +More info in [AWS SDK for Java](https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/credentials.html) and +[Boto3 Docs](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html) + +### 3. Restart both servers + +After updating the configuration files, we are ready to restart both services. When the OM server starts, it will +automatically detect that a Secrets Manager has been configured and will migrate all our sensitive data and remove it +from our DB. + +If everything goes as planned, all the data would be displayed using the secrets names which starts with +`/openmetadata/...` in your AWS Secrets Manager console. The following image shows what it should look like: + +

+secrets-manager-console +

+ +**Note:** If we want to change the starting path for our secrets names from `openmetadata` to a different one, we have +to change the property `clusterName` in our `openmetadata.yaml` + +## CLI + +After enabling the Secret Manager, we also have to make a slight change in our workflows YAML files. In the +`workflowConfig` we have to add the secret manager configuration: + +```yaml +workflowConfig: + openMetadataServerConfig: + secretsManagerProvider: aws + secretsManagerCredentials: + awsAccessKeyId: + awsSecretAccessKey: + awsRegion: + hostPort: + authProvider: +``` + diff --git a/openmetadata-docs/content/deployment/secrets-manager/supported-implementations/aws-ssm-parameter-store/index.md b/openmetadata-docs/content/deployment/secrets-manager/supported-implementations/aws-ssm-parameter-store/index.md new file mode 100644 index 00000000000..a72cc4aa7c1 --- /dev/null +++ b/openmetadata-docs/content/deployment/secrets-manager/supported-implementations/aws-ssm-parameter-store/index.md @@ -0,0 +1,114 @@ +--- +title: AWS Systems Manager Parameter Store +slug: /deployment/secrets-manager/supported-implementations/aws-ssm-parameter-store +--- + +# AWS Systems Manager Parameter Store + +The setup steps covers the use of the managed version of the AWS Systems Manager Parameter Store as secrets manager but +for the non-managed follow only the steps related to the Airflow server and CLI. + +## Setup + +### 1. Permissions needed + +These are the permissions required in the IAM policy to enable the AWS Systems Manager Parameter Store in OpenMetadata. + +```json +{ + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Action": [ + "ssm:PutParameter", + "ssm:GetParameter" + ], + "Resource": "*" + } + ] +} +``` + +### 2. Update configuration + +We have to set up the secret manager provider we want to use, that in our case is `aws-ssm`, and the credentials for our +AWS account. + +The changes to be done in `openmetadata.yaml` file of the OpenMetadata server are: + +```yaml +... +secretsManagerConfiguration: + secretsManager: managed-aws-ssm # or env var SECRET_MANAGER. For non-managed use 'aws-ssm'. + parameters: + region: # or env var OM_SM_REGION + accessKeyId: # or env var OM_SM_ACCESS_KEY_ID + secretAccessKey: # or env var OM_SM_ACCESS_KEY +... +``` + +And these are the changes required in `airflow.cfg` of our Airflow instance: + +```properties +... +[openmetadata_secrets_manager] +aws_region = +aws_access_key_id = +aws_secret_access_key = +... +``` + +As an alternative to editing the `airflow.cfg` file, we can also set the following environment variables: + +```bash +AIRFLOW__OPENMETADATA_SECRETS_MANAGER__AWS_REGION= +AIRFLOW__OPENMETADATA_SECRETS_MANAGER__AWS_ACCESS_KEY_ID= +AIRFLOW__OPENMETADATA_SECRETS_MANAGER__AWS_SECRET_ACCESS_KEY= +``` + +If no parameters are provided for the AWS account, or only ``, it will use the default credentials. The +default credential will look for credentials in: + +1. **Environment variables** - `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`. +2. **Shared credential file** - `~/.aws/credentials` +3. **AWS config file** - `~/.aws/config` +4. **Assume Role provider** +5. Instance metadata service on an Amazon EC2 instance that has an IAM role configured + +More info in [AWS SDK for Java](https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/credentials.html) and +[Boto3 Docs](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html) + +### 3. Restart both servers + +After updating the configuration files, we are ready to restart both services. When the OM server starts, it will +automatically detect that a Secrets Manager has been configured and will migrate all our sensitive data and remove it +from our DB. + +If everything goes as planned, all the data would be displayed using the parameters names which starts with +`/openmetadata/...` in your AWS Systems Manager Parameter Store console. The following image shows what it should look +like: + +

+ssm-parameter-store-console +

+ +**Note:** If we want to change the starting path for our secrets names from `openmetadata` to a different one, we have +to change the property `clusterName` in our `openmetadata.yaml` + +## CLI + +After enabling the Secret Manager, we also have to make a slight change in our workflows YAML files. In the +`workflowConfig` we have to add the secret manager configuration: + +```yaml +workflowConfig: + openMetadataServerConfig: + secretsManagerProvider: aws-ssm + secretsManagerCredentials: + awsAccessKeyId: + awsSecretAccessKey: + awsRegion: + hostPort: + authProvider: +``` \ No newline at end of file diff --git a/openmetadata-docs/content/deployment/secrets-manager/supported-implementations/index.md b/openmetadata-docs/content/deployment/secrets-manager/supported-implementations/index.md new file mode 100644 index 00000000000..f2f45504ad6 --- /dev/null +++ b/openmetadata-docs/content/deployment/secrets-manager/supported-implementations/index.md @@ -0,0 +1,27 @@ +--- +title: Secrets Manager +slug: /deployment/secrets-manager/supported-implementations +--- + +# Supported implementations + +This is our list of supported Secrets Manager implementations: + + + + AWS Secrets Manager + + + AWS Systems Manager Parameter Store + + \ No newline at end of file diff --git a/openmetadata-docs/content/menu.md b/openmetadata-docs/content/menu.md index 58bddc76e24..65dd6b48e01 100644 --- a/openmetadata-docs/content/menu.md +++ b/openmetadata-docs/content/menu.md @@ -134,6 +134,17 @@ site_menu: - category: Deployment / Enable Security / Enable JWT Tokens url: /deployment/security/enable-jwt-tokens + - category: Deployment / Enable Secrets Manager + url: /deployment/secrets-manager + - category: Deployment / Enable Secrets Manager / Supported Implementations + url: /deployment/secrets-manager/supported-implementations + - category: Deployment / Enable Secrets Manager / Supported Implementations / AWS Secrets Manager + url: /deployment/secrets-manager/supported-implementations/aws-secrets-manager + - category: Deployment / Enable Secrets Manager / Supported Implementations / AWS SSM Parameter Store + url: /deployment/secrets-manager/supported-implementations/aws-ssm-parameter-store + - category: Deployment / Enable Secrets Manager / How to add a new implementation + url: /deployment/secrets-manager/how-to-add-a-new-implementation + - category: Deployment / Upgrade OpenMetadata url: /deployment/upgrade - category: Deployment / Upgrade OpenMetadata / Upgrade on Bare Metal diff --git a/openmetadata-spec/src/main/resources/json/schema/entity/services/connections/metadata/secretsManagerProvider.json b/openmetadata-spec/src/main/resources/json/schema/entity/services/connections/metadata/secretsManagerProvider.json index b40a3cbb79b..0f0be2e1ef4 100644 --- a/openmetadata-spec/src/main/resources/json/schema/entity/services/connections/metadata/secretsManagerProvider.json +++ b/openmetadata-spec/src/main/resources/json/schema/entity/services/connections/metadata/secretsManagerProvider.json @@ -1,5 +1,5 @@ { - "$id": "https://open-metadata.org/schema/entity/services/connections/metadata/secretsManagersProvider.json", + "$id": "https://open-metadata.org/schema/entity/services/connections/metadata/secretsManagerProvider.json", "$schema": "http://json-schema.org/draft-07/schema#", "title": "Secrets Manager Provider", "description": "OpenMetadata Secrets Manager Provider. Make sure to configure the same secrets manager providers as the ones configured on the OpenMetadata server.",