Modify Docs related to SSL (#7927)

This commit is contained in:
Ayush Shah 2022-10-04 16:58:46 +05:30 committed by GitHub
parent f77f0c8130
commit 65525316d7
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
8 changed files with 190 additions and 1 deletions

View File

@ -449,7 +449,41 @@ metadata profile -c <path-to-yaml>
``` ```
Note how instead of running `ingest`, we are using the `profile` command to select the Profiler workflow. Note how instead of running `ingest`, we are using the `profile` command to select the Profiler workflow.
## SSL Configuration
In order to integrate SSL in the Metadata Ingestion Config, the user will have to add the SSL config under connectionArguments which is placed in the source.
```yaml
---
source:
type: mariadb
serviceName: "<service name>"
serviceConnection:
config:
type: MariaDB
username: <username>
password: <password>
hostPort: <hostPort>
...
...
connectionArguments:
ssl:
ssl_ca: /path/to/client-ssl/ca.pem,
ssl_cert: /path/to/client-ssl/client-cert.pem
ssl_key: /path/to/client-ssl/client-key.pem
#ssl_disabled: True #boolean
#ssl_verify_cert: True #boolean
#ssl_verify_identity: True #boolean
```
- **ssl**: A dict of arguments which contains:
- **ssl_ca**: Path to the file that contains a PEM-formatted CA certificate.
- **ssl_cert**: Path to the file that contains a PEM-formatted client certificate.
- **ssl_disabled**: A boolean value that disables usage of TLS.
- **ssl_key**: Path to the file that contains a PEM-formatted private key for the client certificate.
- **ssl_verify_cert**: Set to true to check the server certificate's validity.
- **ssl_verify_identity**: Set to true to check the server's identity.
## DBT Integration ## DBT Integration
You can learn more about how to ingest DBT models' definitions and their lineage [here](https://docs.open-metadata.org/openmetadata/ingestion/workflows/metadata/dbt). You can learn more about how to ingest DBT models' definitions and their lineage [here](https://docs.open-metadata.org/openmetadata/ingestion/workflows/metadata/dbt).

View File

@ -168,6 +168,17 @@ caption="Configure Metadata Ingestion Page"
- **Mark Deleted Tables (toggle)**: Set the Mark Deleted Tables toggle to flag tables as soft-deleted if they are not present anymore in the source system. - **Mark Deleted Tables (toggle)**: Set the Mark Deleted Tables toggle to flag tables as soft-deleted if they are not present anymore in the source system.
- **Mark Deleted Tables from Filter Only (toggle)**: Set the Mark Deleted Tables from Filter Only toggle to flag tables as soft-deleted if they are not present anymore within the filtered schema or database only. This flag is useful when you have more than one ingestion pipelines. For example if you have a schema - **Mark Deleted Tables from Filter Only (toggle)**: Set the Mark Deleted Tables from Filter Only toggle to flag tables as soft-deleted if they are not present anymore within the filtered schema or database only. This flag is useful when you have more than one ingestion pipelines. For example if you have a schema
#### **SSL Configuration**
In order to integrate SSL in the Metadata Ingestion Config, the user will have to add the SSL config under connectionArguments which is placed in the source.
- **ssl**: A dict of arguments which contains:
- **ssl_ca**: Path to the file that contains a PEM-formatted CA certificate.
- **ssl_cert**: Path to the file that contains a PEM-formatted client certificate.
- **ssl_disabled**: A boolean value that disables usage of TLS.
- **ssl_key**: Path to the file that contains a PEM-formatted private key for the client certificate.
- **ssl_verify_cert**: Set to true to check the server certificate's validity.
- **ssl_verify_identity**: Set to true to check the server's identity.
### 7. Schedule the Ingestion and Deploy ### 7. Schedule the Ingestion and Deploy
Scheduling can be set up at an hourly, daily, or weekly cadence. The Scheduling can be set up at an hourly, daily, or weekly cadence. The

View File

@ -449,6 +449,41 @@ metadata profile -c <path-to-yaml>
``` ```
Note how instead of running `ingest`, we are using the `profile` command to select the Profiler workflow. Note how instead of running `ingest`, we are using the `profile` command to select the Profiler workflow.
## SSL Configuration
In order to integrate SSL in the Metadata Ingestion Config, the user will have to add the SSL config under connectionArguments which is placed in the source.
```yaml
---
source:
type: mysql
serviceName: "<service name>"
serviceConnection:
config:
type: Mysql
username: <username>
password: <password>
hostPort: <hostPort>
...
...
connectionArguments:
ssl:
ssl_ca: /path/to/client-ssl/ca.pem,
ssl_cert: /path/to/client-ssl/client-cert.pem
ssl_key: /path/to/client-ssl/client-key.pem
#ssl_disabled: True #boolean
#ssl_verify_cert: True #boolean
#ssl_verify_identity: True #boolean
```
- **ssl**: A dict of arguments which contains:
- **ssl_ca**: Path to the file that contains a PEM-formatted CA certificate.
- **ssl_cert**: Path to the file that contains a PEM-formatted client certificate.
- **ssl_disabled**: A boolean value that disables usage of TLS.
- **ssl_key**: Path to the file that contains a PEM-formatted private key for the client certificate.
- **ssl_verify_cert**: Set to true to check the server certificate's validity.
- **ssl_verify_identity**: Set to true to check the server's identity.
## DBT Integration ## DBT Integration

View File

@ -170,6 +170,18 @@ caption="Configure Metadata Ingestion Page"
- **Mark Deleted Tables (toggle)**: Set the Mark Deleted Tables toggle to flag tables as soft-deleted if they are not present anymore in the source system. - **Mark Deleted Tables (toggle)**: Set the Mark Deleted Tables toggle to flag tables as soft-deleted if they are not present anymore in the source system.
- **Mark Deleted Tables from Filter Only (toggle)**: Set the Mark Deleted Tables from Filter Only toggle to flag tables as soft-deleted if they are not present anymore within the filtered schema or database only. This flag is useful when you have more than one ingestion pipelines. For example if you have a schema - **Mark Deleted Tables from Filter Only (toggle)**: Set the Mark Deleted Tables from Filter Only toggle to flag tables as soft-deleted if they are not present anymore within the filtered schema or database only. This flag is useful when you have more than one ingestion pipelines. For example if you have a schema
#### **SSL Configuration**
In order to integrate SSL in the Metadata Ingestion Config, the user will have to add the SSL config under connectionArguments which is placed in the source.
- **ssl**: A dict of arguments which contains:
- **ssl_ca**: Path to the file that contains a PEM-formatted CA certificate.
- **ssl_cert**: Path to the file that contains a PEM-formatted client certificate.
- **ssl_disabled**: A boolean value that disables usage of TLS.
- **ssl_key**: Path to the file that contains a PEM-formatted private key for the client certificate.
- **ssl_verify_cert**: Set to true to check the server certificate's validity.
- **ssl_verify_identity**: Set to true to check the server's identity.
### 7. Schedule the Ingestion and Deploy ### 7. Schedule the Ingestion and Deploy
Scheduling can be set up at an hourly, daily, or weekly cadence. The Scheduling can be set up at an hourly, daily, or weekly cadence. The

View File

@ -553,6 +553,44 @@ metadata profile -c <path-to-yaml>
Note how instead of running `ingest`, we are using the `profile` command to select the Profiler workflow. Note how instead of running `ingest`, we are using the `profile` command to select the Profiler workflow.
## SSL Configuration
In order to integrate SSL in the Metadata Ingestion Config, the user will have to add the SSL config under connectionArguments which is placed in the source.
```yaml
---
source:
type: redshift
serviceName: <service name>
serviceConnection:
config:
type: Redshift
hostPort: cluster.name.region.redshift.amazonaws.com:5439
username: username
...
...
...
connectionArguments:
sslmode: <ssl-mode>
```
### SSL Modes
There are couple of types of SSL modes that Redshift supports which can be added to ConnectionArguments, they are as follows:
- **disable**: SSL is disabled and the connection is not encrypted.
- **allow**: SSL is used if the server requires it.
- **prefer**: SSL is used if the server supports it. Amazon Redshift supports SSL, so SSL is used when you set sslmode to prefer.
- **require**: SSL is required.
- **verify-ca**: SSL must be used and the server certificate must be verified.
- **verify-full**: SSL must be used. The server certificate must be verified and the server hostname must match the hostname attribute on the certificate.
For more information, you can visit [Redshift SSL documentation](https://docs.aws.amazon.com/redshift/latest/mgmt/connecting-ssl-support.html)
## DBT Integration ## DBT Integration
You can learn more about how to ingest DBT models' definitions and their lineage [here](https://docs.open-metadata.org/openmetadata/ingestion/workflows/metadata/dbt). You can learn more about how to ingest DBT models' definitions and their lineage from [here](https://docs.open-metadata.org/openmetadata/ingestion/workflows/metadata/dbt).

View File

@ -177,6 +177,26 @@ caption="Configure Metadata Ingestion Page"
</Note> </Note>
#### **SSL Configuration**
In order to integrate SSL in the Metadata Ingestion Config, the user will have to add the SSL config under connectionArguments which is placed in the source.
##### **SSL Modes**
There are couple of types of SSL modes that Redshift supports which can be added to ConnectionArguments, they are as follows:
- **disable**: SSL is disabled and the connection is not encrypted.
- **allow**: SSL is used if the server requires it.
- **prefer**: SSL is used if the server supports it. Amazon Redshift supports SSL, so SSL is used when you set sslmode to prefer.
- **require**: SSL is required.
- **verify-ca**: SSL must be used and the server certificate must be verified.
- **verify-full**: SSL must be used. The server certificate must be verified and the server hostname must match the hostname attribute on the certificate.
For more information, you can visit [Redshift SSL documentation](https://docs.aws.amazon.com/redshift/latest/mgmt/connecting-ssl-support.html)
### 7. Schedule the Ingestion and Deploy ### 7. Schedule the Ingestion and Deploy
Scheduling can be set up at an hourly, daily, or weekly cadence. The Scheduling can be set up at an hourly, daily, or weekly cadence. The

View File

@ -463,6 +463,34 @@ metadata profile -c <path-to-yaml>
``` ```
Note how instead of running `ingest`, we are using the `profile` command to select the Profiler workflow. Note how instead of running `ingest`, we are using the `profile` command to select the Profiler workflow.
## SSL Configuration
In order to integrate SSL in the Metadata Ingestion Config, the user will have to add the SSL config under **connectionArguments** which is placed in source.
```yaml
---
source:
type: trino
serviceName: <service name>
serviceConnection:
config:
type: Trino
hostPort: <hostPort>
username: <username>
catalog: <catalog>
...
...
connectionArguments:
verify: <verify-mode>
```
### SSL Modes
There are couple of types of SSL modes that redshift supports which can be added to ConnectionArguments, they are as follows:
- **false**: In order to disable SSL verification, set the `verify` parameter to `False`.
- **\<path-to-crt\>**: To use self-signed certificates, specify a path to the certificate in `verify` parameter.
More details can be found in [the Python requests library documentation](https://requests.readthedocs.io/en/latest/user/advanced/#ssl-cert-verification).
## DBT Integration ## DBT Integration

View File

@ -169,6 +169,17 @@ caption="Configure Metadata Ingestion Page"
- **Mark Deleted Tables (toggle)**: Set the Mark Deleted Tables toggle to flag tables as soft-deleted if they are not present anymore in the source system. - **Mark Deleted Tables (toggle)**: Set the Mark Deleted Tables toggle to flag tables as soft-deleted if they are not present anymore in the source system.
- **Mark Deleted Tables from Filter Only (toggle)**: Set the Mark Deleted Tables from Filter Only toggle to flag tables as soft-deleted if they are not present anymore within the filtered schema or database only. This flag is useful when you have more than one ingestion pipelines. For example if you have a schema - **Mark Deleted Tables from Filter Only (toggle)**: Set the Mark Deleted Tables from Filter Only toggle to flag tables as soft-deleted if they are not present anymore within the filtered schema or database only. This flag is useful when you have more than one ingestion pipelines. For example if you have a schema
#### **SSL Configuration**
In order to integrate SSL in the Metadata Ingestion Config, the user will have to add the SSL config under **connectionArguments** which is placed in source.
##### **SSL Modes**
There are couple of types of SSL modes that redshift supports which can be added to ConnectionArguments, they are as follows:
- **false**: In order to disable SSL verification, set the `verify` parameter to `False`.
- **\<path-to-crt\>**: To use self-signed certificates, specify a path to the certificate in `verify` parameter.
More details can be found in [the Python requests library documentation](https://requests.readthedocs.io/en/latest/user/advanced/#ssl-cert-verification).
### 7. Schedule the Ingestion and Deploy ### 7. Schedule the Ingestion and Deploy
Scheduling can be set up at an hourly, daily, or weekly cadence. The Scheduling can be set up at an hourly, daily, or weekly cadence. The