Docs structure and migrations (#5999)

Docs structure and migrations (#5999)
This commit is contained in:
Pere Miquel Brull 2022-07-11 15:19:11 +02:00 committed by GitHub
parent 625b30ab94
commit 6c8adb8014
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
43 changed files with 424 additions and 44 deletions

View File

@ -5,3 +5,9 @@ slug: /deployment/airflow
# Airflow Deployment
## Custom Airflow Installation
## Configure in the OpenMetadata Server
Show all security info here

View File

@ -0,0 +1,6 @@
---
title: Enable JWT Tokens
slug: /deployment/security/enable-jwt-tokens
---
# Enable JWT Tokens

View File

@ -0,0 +1,27 @@
---
title: Enable SSL
slug: /deployment/security/enable-ssl
---
# Enable SSL
In this section we will guide you through adding SSL to your OpenMetadata deployment with two different approaches:
<InlineCalloutContainer>
<InlineCallout
color="violet-70"
bold="Use Nginx"
icon="add_moderator"
href="/deployment/security/enable-ssl/nginx"
>
Use Nginx to enable SSL. This is the simplest solution.
</InlineCallout>
<InlineCallout
color="violet-70"
bold="Azure SSO"
icon="add_moderator"
href="/deployment/security/enable-ssl/jwt"
>
Set SSL directly at the OpenMetadata server.
</InlineCallout>
</InlineCalloutContainer>

View File

@ -0,0 +1,111 @@
---
title: Enable SSL with Nginx
slug: /deployment/security/enable-ssl/nginx
---
# Enable SSL with Nginx
Nginx can be used as a load balancer or an SSL termination point for OpenMetadata.
In this section, we will look at how to use Nginx and Certbot to deploy SSL. The below instructions are for Ubuntu 20
and any other flavor of Linux please find similar instructions.
## Install Nginx
Nginx can be installed to a completely different host where you are running OpenMetadata Server or on the same host.
For simplicity, we will do this on the same host as the OpenMetadata server.
```commandline
sudo apt update
sudo apt install nginx
sudo systemctl start nginx
```
## Configure Nginx to redirect requests to OpenMetadata
For Nginx to serve this content, its necessary to create a server block with the correct directives.
Instead of modifying the default configuration file directly, lets make a new one at `/etc/nginx/sites-available/openmetadata`:
```commandline
sudo vi /etc/nginx/sites-available/openmetadata
```
And add the below content
```commandline
server {
access_log /var/log/nginx/sandbox-access.log;
error_log /var/log/nginx/sandbox-error.log;
server_name sandbox.open-metadata.org;
location / {
proxy_pass http://127.0.0.1:8585;
}
}
```
In the above configuration, please ensure that the `server_name` matches the domain where you are hosting the OpenMetadata
server. Also, the `proxy_pass` configuration should point to the OpenMetadata server port.
Then, link the configuration to `sites-enabled` and restart nginx:
```commandline
sudo ln -s /etc/nginx/sites-available/openmetadata /etc/nginx/sites-enabled/openmetadata
sudo systemctl restart nginx
```
The above configuration will serve at port 80, so if you configured a domain like `sandbox.open-metadata.org` one can
start accessing OpenMetadata server by just pointing the browser to [http://sandbox.open-metadata.org](http://sandbox.open-metadata.org).
## Enable SSL using Certbot
Certbot, [https://certbot.eff.org/](https://certbot.eff.org/), is a non-profit org that distributes the certified X509
certs and renews them as well.
```commandline
sudo apt install certbot python3-certbot-nginx
sudo systemctl reload nginx
```
## Obtaining an SSL Certificate
Certbot provides a variety of ways to obtain SSL certificates through plugins. The Nginx plugin will take care of
reconfiguring Nginx and reloading the config whenever necessary. To use this plugin, type the following:
```commandline
sudo certbot --nginx -d sandbox.open-metadata.org
```
Replace` sandbox.open-metadata.org` with your domain for OpenMetadata.
If this is your first time running certbot, you will be prompted to enter an email address and agree to the terms of
service. After doing so, certbot will communicate with the `Let's Encrypt` server, then run a challenge to verify that
you control the domain youre requesting a certificate for.
If thats successful, certbot will ask how youd like to configure your HTTPS settings.
## Verifying Certbot Auto-Renewal
`Let's Encrypt`'s certificates are only valid for ninety days. This is to encourage users to automate their certificate
renewal process. The certbot package we installed takes care of this for us by adding a `systemd` timer that will run
twice a day and automatically renew any certificate thats within thirty days of expiration.
You can query the status of the timer with `systemctl`:
```commandline
sudo systemctl status certbot.timer
```
to renew, you can run the following command
```commandline
sudo certbot renew --dry-run
```
## Summary
In this tutorial, we walked through the setup of Nginx to serve the requests to OpenMetadata and used Certbot to enable
SSL on Nginx.
Do keep in mind that we secured the external connection to Nginx, and Nginx terminates the SSL connections,
and the rest of the transport Nginx to the OpenMetadata server is on Plaintext. However, OpenMetadata server should be
configured to listen to only localhost requests, i.e., It cannot be reached directly from outside traffic except for
Nginx on that host. This makes it a secure SSL.

View File

@ -0,0 +1,93 @@
---
title: Enable SSL at the OpenMetadata Server
slug: /deployment/security/enable-ssl/openmetadata-server
---
# Enable SSL at the OpenMetadata Server
The OpenMetadata Server is built using **Dropwizard** and **Jetty**. In this section, we will go through the steps
involved in setting up SSL for Jetty.
If you would like a simple way to set up SSL, please refer to the guide using [Nginx](/deployment/security/enable-ssl/nginx).
However, this step can be treated as an additional layer of adding SSL to OpenMetadata. In cases where one would use
Nginx as a load balancer or AWS LB, you can set up SSL at the OpenMetadata server level such that traffic from the
load balancer to OpenMetadata is going through an encrypted channel.
## Create Self-Signed Certificate
A self-signed certificate should only be used for POC (demo) or `localhost` installation.
For production scenarios, please reach out to your DevOps team to issue an X509 certificate which you can import into a
Keystore. Run the below command to generate an X509 Certificate and import it into keystore:
```commandline
keytool -keystore openmetadata.keystore.jks -alias localhost -keyalg RSA -keysize 2048 -sigalg SHA256withRSA -genkey -validity 365
```
<Image src="/images/deployment/security/enable-ssl/openmetadata-server/keystore-1.png" alt="keystore"/>
For this example, we are configuring the password to be `test12`. Copy the generated `openmetadata.keystore.jks` to
OpenMetadata installation path under the `conf` directory.
<Image src="/images/deployment/security/enable-ssl/openmetadata-server/keystore-2.png" alt="keystore"/>
## Configure openmetadata.yaml
Add the below section to your `openmetadata.yaml` under the `conf` directory. Please add the password you set for the
Keystore generated above in the config below.
```yaml
server:
rootPath: '/api/*'
applicationConnectors:
- type: https
port: ${SERVER_PORT:-8585}
keyStorePath: ./conf/openmetadata.keystore.jks
keyStorePassword: test12
keyStoreType: JKS
supportedProtocols: [TLSv1.2, TLSv1.3]
excludedProtocols: [SSL, SSLv2, SSLv2Hello, SSLv3]
```
## Access OpenMetadata server in the browser
These steps are not necessary if you used proper X509 certificated signed by trusted CA Authority.
Since we used self-signed certificates, browsers such as Chrome or Brave will not allow you to visit
[https://localhost:8585](https://localhost:8585). You'll get the following error page and there is no way to proceed.
<Image src="/images/deployment/security/enable-ssl/openmetadata-server/browser.png" alt="browser"/>
However, the Safari browser allows you to visit if you click advanced and click proceed. To work around this issue, on
OS X, you can import the certificate into the keychain and trust it so that browsers can trust and allow you to access
OpenMetadata.
### Export X509 certificate from Keystore
Run the below command to export the X509 cert.
```commandline
keytool -export -alias localhost -keystore openmetadata.keystore.jks -rfc -file public.cert
```
### Import public cert into Keychain - OS X only
Open the KeyChain app in OS X, drag and drop the `public.cert` file generated in the previous command into the Keychain:
<Image src="/images/deployment/security/enable-ssl/openmetadata-server/import-1.png" alt="import"/>
Double-click on `localhost`:
<Image src="/images/deployment/security/enable-ssl/openmetadata-server/import-2.png" alt="import"/>
Click on `Trust` to open and set `Always Trust`:
<Image src="/images/deployment/security/enable-ssl/openmetadata-server/import-3.png" alt="import"/>
Once the above steps are finished, all the browsers will allow you to visit the OpenMetadata server using HTTPS.
However, you'll still a warning in the address bar. All of these steps are not necessary with an X509 certificate issued
by a trusted authority and one should always use that in production.

View File

@ -0,0 +1,6 @@
---
title: Build the Code and Run Tests
slug: /developers/build-code-and-run-tests
---
# Build the Code and Run Tests

View File

@ -3,7 +3,7 @@ title: Contribute
slug: /developers/contribute
---
# Contribute to Open Metadatas
# How to Contribute
## Local Setup

View File

@ -1,6 +1,6 @@
---
title: OpenMetadata SDK
slug: /developers/sdk
title: OpenMetadata
slug: /developers
---
# OpenMetadata SDK
# OpenMetadata

View File

@ -1,6 +0,0 @@
---
title: Ingestion Workflows
slug: /developers/ingestion-workflows
---
# Ingestion Workflows

View File

@ -0,0 +1,6 @@
---
title: Open Source Community
slug: /developers/open-source-community
---
# Open Source Community

View File

@ -5,7 +5,7 @@ slug: /main-concepts
# Main Concepts
Here yiou can find out some information about the main concepts around Open Metadata
Here you can find out some information about the main concepts around Open Metadata
## Why Open Metadata?

View File

@ -1,6 +1,6 @@
---
title: API's
title: APIs
slug: /main-concepts/metadata-standard/apis
---
# API's
# APIs

View File

@ -1,6 +0,0 @@
---
title: Entities
slug: /main-concepts/metadata-standard/entities
---
# Entities

View File

@ -0,0 +1,6 @@
---
title: Schemas
slug: /main-concepts/metadata-standard/schemas
---
# Schemas

View File

@ -85,6 +85,14 @@ site_menu:
url: /deployment/security/okta/bare-metal
- category: Deployment / Enable Security / Okta SSO / Kubernetes
url: /deployment/security/okta/kubernetes
- category: Deployment / Enable Security / Enable SSL
url: /deployment/security/enable-ssl
- category: Deployment / Enable Security / Enable SSL / Use Nginx
url: /deployment/security/enable-ssl/nginx
- category: Deployment / Enable Security / Enable SSL / Use the OpenMetadata Server
url: /deployment/security/enable-ssl/openmetadata-server
- category: Deployment / Enable Security / Enable JWT Tokens
url: /deployment/security/enable-jwt-tokens
- category: Deployment / Upgrade OpenMetadata
url: /deployment/upgrade
@ -336,8 +344,7 @@ site_menu:
- category: OpenMetadata / Discovery & Collaboration
url: /openmetadata/discovery-collaboration
- category: OpenMetadata / Teams & Users
url: /openmetadata/users
- category: OpenMetadata / Data Quality
url: /openmetadata/data-quality
- category: OpenMetadata / Data Quality / Tests
@ -345,37 +352,64 @@ site_menu:
- category: OpenMetadata / Data Quality / Metrics
url: /openmetadata/data-quality/metrics
- category: OpenMetadata / Teams & Users
url: /openmetadata/users
- category: OpenMetadata / Integrations
url: /openmetadata/integrations
- category: OpenMetadata / Integrations / Great Expectations & OpenMetadata
url: /openmetadata/integrations/great-expectations
- category: OpenMetadata / Integrations / Prefect
url: /openmetadata/integrations/prefect
- category: Main Concepts
url: /main-concepts
color: violet-70
icon: developer_board
- category: Main Concepts / Get started
url: /main-concepts/get-started
icon: public
- category: Main Concepts / High Level Design
url: /main-concepts/high-level-design
- category: Main Concepts / Metadata Standard
url: /main-concepts/metadata-standard
- category: Main Concepts / Metadata Standard / Entities
url: /main-concepts/metadata-standard/entities
- category: Main Concepts / Metadata Standard / API's
- category: Main Concepts / Metadata Standard / Schemas
url: /main-concepts/metadata-standard/schemas
- category: Main Concepts / Metadata Standard / APIs
url: /main-concepts/metadata-standard/apis
- category: Developers
url: /developers
color: violet-70
icon: developer_mode
- category: Developers / Open Source Community
url: /developers/open-source-community
- category: Developers / Contribute
url: /developers/contribute
- category: Developers / Ingestion Workflows
url: /developers/ingestion-workflows
- category: Developers / Webhooks
url: /developers/webhooks
- category: Developers / OpenMetadata SDK
url: /developers/sdk
- category: Developers / OpenMetadata SDK / Python SDK
url: /developers/sdk/python
- category: Developers / OpenMetadata SDK / Python SDK / ML Model Python SDK
url: /developers/sdk/python/ml-model-sdk
- category: Developers / OpenMetadata SDK / Python SDK / Build a Connector
url: /developers/sdk/python/build-connector
- category: SDK
url: /sdk
color: violet-70
icon: developer_board
- category: SDK / Python SDK
url: /sdk/python
- category: SDK / Python SDK / Entities
url: /sdk/python/entities
- category: SDK / Python SDK / Entities / ML Model Python SDK
url: /sdk/python/entities/ml-model-sdk
- category: SDK / Python SDK / Ingestion
url: /sdk/python/ingestion
- category: SDK / Python SDK / Ingestion / DBT
url: /sdk/python/ingestion/dbt
- category: SDK / Python SDK / Build a Connector
url: /sdk/python/build-connector
- category: SDK / Python SDK / Build a Connector / Source
url: /sdk/python/build-connector/source
- category: SDK / Python SDK / Build a Connector / Stage
url: /sdk/python/build-connector/stage
- category: SDK / Python SDK / Build a Connector / Sink
url: /sdk/python/build-connector/sink
- category: SDK / Python SDK / Build a Connector / Bulk Sink
url: /sdk/python/build-connector/bulk-sink
- category: SDK / Java SDK
url: /sdk/java
---

View File

@ -4,3 +4,13 @@ slug: /openmetadata/discovery-collaboration
---
# Discovery & Collaboration
## Search
## Threads
## Tasks
## Glossaries
## Tags

View File

@ -1,9 +1,9 @@
---
title: OpenMetadata UI
title: OpenMetadata
slug: /openmetadata
---
# OpenMetadata UI
# OpenMetadata
The OpenMetadata UI is the single stop for all users in the organisation to discover and collaborate.

View File

@ -4,3 +4,11 @@ slug: /openmetadata/ingestion
---
# Metadata Ingestion
Explain how we have different types of workflows and the metadata
that we can ingest automatically:
- e.g., table metadata
- DBT
- Lineage
- Usage

View File

@ -4,3 +4,7 @@ slug: /openmetadata/ingestion/lineage
---
# Entity Lineage
- Automated lineage (Usage workflow + views)
- Manual Lineage
- Tools we use

View File

@ -4,3 +4,5 @@ slug: /openmetadata/ingestion/workflows
---
# Ingestion Workflows
Explain ingestion, usage and profiler workflows

View File

@ -0,0 +1,6 @@
---
title: Great Expectations & OpenMetadata
slug: /openmetadata/integrations/great-expectations
---
## GE

View File

@ -0,0 +1,8 @@
---
title: Integrations
slug: /openmetadata/integrations
---
## GE
## Prefect

View File

@ -0,0 +1,7 @@
---
title: Prefect & OpenMetadata
slug: /openmetadata/integrations/prefect
---
## Prefect

View File

@ -5,3 +5,4 @@ slug: /openmetadata/users
# Teams & Users
## Roles and Policies

View File

@ -1,6 +1,6 @@
---
title: OpenMetadata SDK
slug: /developers/sdk
slug: /sdk
---
# OpenMetadata SDK

View File

@ -0,0 +1,6 @@
---
title: Java SDK
slug: /sdk/java
---
# Java SDK

View File

@ -0,0 +1,6 @@
---
title: Bulk Sink
slug: /sdk/python/build-connector/bulk-sink
---
# Bulk Sink

View File

@ -1,6 +1,6 @@
---
title: Build a Connector
slug: /developers/sdk/python/build-connector
slug: /sdk/python/build-connector
---
# Build a Connector

View File

@ -0,0 +1,6 @@
---
title: Sink
slug: /sdk/python/build-connector/sink
---
# Sink

View File

@ -0,0 +1,6 @@
---
title: Source
slug: /sdk/python/build-connector/source
---
# Source

View File

@ -0,0 +1,6 @@
---
title: Stage
slug: /sdk/python/build-connector/stage
---
# Stage

View File

@ -0,0 +1,6 @@
---
title: Python SDK Entities
slug: /sdk/python/entities
---
# Python SDK Entities

View File

@ -1,6 +1,6 @@
---
title: ML Model Python SDK
slug: /developers/sdk/python/ml-model-sdk
slug: /sdk/python/ml-model-sdk
---
# ML Model Python SDK

View File

@ -1,6 +1,6 @@
---
title: Python SDK
slug: /developers/sdk/python
slug: /sdk/python
---
# Python SDK

View File

@ -0,0 +1,7 @@
---
title: Python SDK for DBT
slug: /sdk/python/ingestion/dbt
---
# Python SDK for DBT

View File

@ -0,0 +1,8 @@
---
title: Python SDK Ingestion
slug: /sdk/python/ingestion
---
# Python SDK Ingestion
Snippets and knowledge specific to ingesting metadata using Python.

Binary file not shown.

After

Width:  |  Height:  |  Size: 124 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 135 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 115 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 98 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 127 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 57 KiB