Update setup, ingestion docs

This commit is contained in:
Suresh Srinivas 2021-08-12 11:39:56 -07:00
parent df6030fe12
commit 40911af11f
4 changed files with 111 additions and 13 deletions

View File

@ -37,8 +37,7 @@
## Install
* [Run Locally](install/run-locally.md)
* [Run Docker](install/run-docker.md)
* [Run OpenMetadata](install/run-openmetadata.md)
* [Run in production](install/run-in-production.md)
* [Metadata Ingestion](install/metadata-ingestion/README.md)
* [Ingest Sample Data](install/metadata-ingestion/ingest-sample-data.md)

View File

@ -1,2 +1,21 @@
# Metadata Ingestion
OpenMetadata Ingesiton is a simple framework to build connectors and ingest metadata of various systems through OpenMetadata APIs.
It could be used in an orchestration framework(e.g. Apache Airflow) to ingest metadata.
## Ingest Sample Data
{% page-ref page="ingest-sample-metadata" %}
## Explore and configure connectors
{% page-ref page="connectors" %}
## Run Simple Scheduler
{% page-ref page="scheduler" %}
## Run using Airflow
{% page-ref page="airflow" %}

View File

@ -1,2 +1,58 @@
---
description: >-
This installation doc will help you start a OpenMetadata instances
on your production.
---
# Run in production
Please refer to the previous section [Run Openmetadata](install/run-openmetadata.md) for configuring OpenMetadata.
{% hint style="info" %}
**Prerequisites**
* MySQL >= 8.x
* ElasticSearch >= 7.x
* Airflow or other schedulers to run Ingestion Connectors
{% endhint %}
## Start OpenMetadata
OpenMetadata release ships with ```./bin/openmetadata``` init.d style
script.
```text
cd openmetdata-0.3.0
./bin/openmetdata start
```
We recommend to configure serviced to monitor openmetadata command to restart
incase of any failures.
## Running with a load balancer
One or more OpenMetadata instances can be put behind a load balancer for reverse proxying,
in that case appropriate OpenMetdata url must be mentioned in the load balancer's configuraiton file.
For example, in case Apache mod proxy the VirtualHost tag in the configuration file should be
edited out with the following
```text
<VirtualHost *:80>
<Proxy balancer://mycluster>
BalancerMember http://127.0.0.1:8585 <!-- First OpenMetadata server -->
BalancerMember http://127.0.0.2:8686 <!-- Second OpenMetadata server -->
</Proxy>
ProxyPreserveHost On
ProxyPass / balancer://mycluster/
ProxyPassReverse / balancer://mycluster/
</VirtualHost>
```

View File

@ -4,6 +4,27 @@ description: >-
on your local machine.
---
# Run Docker
[Docker](https://docs.docker.com/get-started/overview/) is an open platform for developing, shipping, and running applications that enables you to separate your applications from your infrastructure so you can deliver software quickly using OS-level virtualization to deliver software in packages called containers.
{% hint style="info" %}
**Prerequisites**
* Docker &gt;= 20.10.x
**Ports to access once the docker is up:**
* OpenMetadata UI: 8585
* Scheduler UI: 7777
{% endhint %}
```text
cd docker/metadata
docker-compose up
```
# Run Locally
{% hint style="success" %}
@ -22,6 +43,18 @@ OpenMetadata is built using Java, DropWizard, Jetty, and MySQL.
{% endhint %}
{% tabs %}
{% tab title="Download the release" %}
Download the latest binary release from [OpenMetadata](https://open-metadata.org/download/), Once you have the tar file,
```bash
# untar it
tar -zxvf openmetadata-0.3.0.tar.gz
# navigate to directory containing the launcher scripts
cd openmetadata-0.3.0
```
{% endtab %}
{% tab title="Build from source " %}
Follow these steps to checkout code from [Github](https://github.com/open-metadata/OpenMetadata) and build OpenMetadata locally
@ -46,17 +79,7 @@ cd openmetadata-1.0.0-SNAPSHOT
```
{% endtab %}
{% tab title="Download the release" %}
Download the latest binary release from [OpenMetadata](https://open-metadata.org/download/), Once you have the tar file,
```bash
# untar it
tar -zxvf openmetadata-1.0.0-SNAPSHOT.tar.gz
# navigate to directory containing the launcher scripts
cd openmetadata-1.0.0-SNAPSHOT
```
{% endtab %}
{% endtabs %}
## Install on your local machine
@ -91,13 +114,14 @@ cd openmetadata-1.0.0-SNAPSHOT
2. Run bootstrap scripts to initiate the database and tables
```text
cd $METADATA_HOME
cd openmetadata-0.3.0
./boostrap/bootstrap-storage.sh migrate
```
3. Start the OpenMetadata Server
```text
cd openmetadata-0.3.0
./bin/openmetadata.sh start
```