mirror of
https://github.com/open-metadata/OpenMetadata.git
synced 2025-12-16 10:08:08 +00:00
Update setup, ingestion docs
This commit is contained in:
parent
df6030fe12
commit
40911af11f
@ -37,8 +37,7 @@
|
|||||||
|
|
||||||
## Install
|
## Install
|
||||||
|
|
||||||
* [Run Locally](install/run-locally.md)
|
* [Run OpenMetadata](install/run-openmetadata.md)
|
||||||
* [Run Docker](install/run-docker.md)
|
|
||||||
* [Run in production](install/run-in-production.md)
|
* [Run in production](install/run-in-production.md)
|
||||||
* [Metadata Ingestion](install/metadata-ingestion/README.md)
|
* [Metadata Ingestion](install/metadata-ingestion/README.md)
|
||||||
* [Ingest Sample Data](install/metadata-ingestion/ingest-sample-data.md)
|
* [Ingest Sample Data](install/metadata-ingestion/ingest-sample-data.md)
|
||||||
|
|||||||
@ -1,2 +1,21 @@
|
|||||||
# Metadata Ingestion
|
# Metadata Ingestion
|
||||||
|
|
||||||
|
OpenMetadata Ingesiton is a simple framework to build connectors and ingest metadata of various systems through OpenMetadata APIs.
|
||||||
|
It could be used in an orchestration framework(e.g. Apache Airflow) to ingest metadata.
|
||||||
|
|
||||||
|
## Ingest Sample Data
|
||||||
|
{% page-ref page="ingest-sample-metadata" %}
|
||||||
|
|
||||||
|
|
||||||
|
## Explore and configure connectors
|
||||||
|
{% page-ref page="connectors" %}
|
||||||
|
|
||||||
|
|
||||||
|
## Run Simple Scheduler
|
||||||
|
|
||||||
|
{% page-ref page="scheduler" %}
|
||||||
|
|
||||||
|
|
||||||
|
## Run using Airflow
|
||||||
|
|
||||||
|
{% page-ref page="airflow" %}
|
||||||
|
|||||||
@ -1,2 +1,58 @@
|
|||||||
|
---
|
||||||
|
description: >-
|
||||||
|
This installation doc will help you start a OpenMetadata instances
|
||||||
|
on your production.
|
||||||
|
---
|
||||||
|
|
||||||
|
|
||||||
# Run in production
|
# Run in production
|
||||||
|
|
||||||
|
Please refer to the previous section [Run Openmetadata](install/run-openmetadata.md) for configuring OpenMetadata.
|
||||||
|
|
||||||
|
{% hint style="info" %}
|
||||||
|
**Prerequisites**
|
||||||
|
|
||||||
|
* MySQL >= 8.x
|
||||||
|
* ElasticSearch >= 7.x
|
||||||
|
* Airflow or other schedulers to run Ingestion Connectors
|
||||||
|
|
||||||
|
{% endhint %}
|
||||||
|
|
||||||
|
|
||||||
|
## Start OpenMetadata
|
||||||
|
|
||||||
|
OpenMetadata release ships with ```./bin/openmetadata``` init.d style
|
||||||
|
script.
|
||||||
|
|
||||||
|
```text
|
||||||
|
cd openmetdata-0.3.0
|
||||||
|
./bin/openmetdata start
|
||||||
|
```
|
||||||
|
|
||||||
|
We recommend to configure serviced to monitor openmetadata command to restart
|
||||||
|
incase of any failures.
|
||||||
|
|
||||||
|
|
||||||
|
## Running with a load balancer
|
||||||
|
|
||||||
|
One or more OpenMetadata instances can be put behind a load balancer for reverse proxying,
|
||||||
|
in that case appropriate OpenMetdata url must be mentioned in the load balancer's configuraiton file.
|
||||||
|
|
||||||
|
For example, in case Apache mod proxy the VirtualHost tag in the configuration file should be
|
||||||
|
edited out with the following
|
||||||
|
|
||||||
|
```text
|
||||||
|
|
||||||
|
<VirtualHost *:80>
|
||||||
|
<Proxy balancer://mycluster>
|
||||||
|
BalancerMember http://127.0.0.1:8585 <!-- First OpenMetadata server -->
|
||||||
|
BalancerMember http://127.0.0.2:8686 <!-- Second OpenMetadata server -->
|
||||||
|
</Proxy>
|
||||||
|
|
||||||
|
ProxyPreserveHost On
|
||||||
|
|
||||||
|
ProxyPass / balancer://mycluster/
|
||||||
|
ProxyPassReverse / balancer://mycluster/
|
||||||
|
</VirtualHost>
|
||||||
|
```
|
||||||
|
|
||||||
@ -4,6 +4,27 @@ description: >-
|
|||||||
on your local machine.
|
on your local machine.
|
||||||
---
|
---
|
||||||
|
|
||||||
|
# Run Docker
|
||||||
|
|
||||||
|
[Docker](https://docs.docker.com/get-started/overview/) is an open platform for developing, shipping, and running applications that enables you to separate your applications from your infrastructure so you can deliver software quickly using OS-level virtualization to deliver software in packages called containers.
|
||||||
|
|
||||||
|
{% hint style="info" %}
|
||||||
|
**Prerequisites**
|
||||||
|
|
||||||
|
* Docker >= 20.10.x
|
||||||
|
|
||||||
|
**Ports to access once the docker is up:**
|
||||||
|
|
||||||
|
* OpenMetadata UI: 8585
|
||||||
|
* Scheduler UI: 7777
|
||||||
|
{% endhint %}
|
||||||
|
|
||||||
|
```text
|
||||||
|
cd docker/metadata
|
||||||
|
docker-compose up
|
||||||
|
```
|
||||||
|
|
||||||
|
|
||||||
# Run Locally
|
# Run Locally
|
||||||
|
|
||||||
{% hint style="success" %}
|
{% hint style="success" %}
|
||||||
@ -22,6 +43,18 @@ OpenMetadata is built using Java, DropWizard, Jetty, and MySQL.
|
|||||||
{% endhint %}
|
{% endhint %}
|
||||||
|
|
||||||
{% tabs %}
|
{% tabs %}
|
||||||
|
{% tab title="Download the release" %}
|
||||||
|
Download the latest binary release from [OpenMetadata](https://open-metadata.org/download/), Once you have the tar file,
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# untar it
|
||||||
|
tar -zxvf openmetadata-0.3.0.tar.gz
|
||||||
|
|
||||||
|
# navigate to directory containing the launcher scripts
|
||||||
|
cd openmetadata-0.3.0
|
||||||
|
```
|
||||||
|
{% endtab %}
|
||||||
|
|
||||||
{% tab title="Build from source " %}
|
{% tab title="Build from source " %}
|
||||||
Follow these steps to checkout code from [Github](https://github.com/open-metadata/OpenMetadata) and build OpenMetadata locally
|
Follow these steps to checkout code from [Github](https://github.com/open-metadata/OpenMetadata) and build OpenMetadata locally
|
||||||
|
|
||||||
@ -46,17 +79,7 @@ cd openmetadata-1.0.0-SNAPSHOT
|
|||||||
```
|
```
|
||||||
{% endtab %}
|
{% endtab %}
|
||||||
|
|
||||||
{% tab title="Download the release" %}
|
|
||||||
Download the latest binary release from [OpenMetadata](https://open-metadata.org/download/), Once you have the tar file,
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# untar it
|
|
||||||
tar -zxvf openmetadata-1.0.0-SNAPSHOT.tar.gz
|
|
||||||
|
|
||||||
# navigate to directory containing the launcher scripts
|
|
||||||
cd openmetadata-1.0.0-SNAPSHOT
|
|
||||||
```
|
|
||||||
{% endtab %}
|
|
||||||
{% endtabs %}
|
{% endtabs %}
|
||||||
|
|
||||||
## Install on your local machine
|
## Install on your local machine
|
||||||
@ -91,13 +114,14 @@ cd openmetadata-1.0.0-SNAPSHOT
|
|||||||
2. Run bootstrap scripts to initiate the database and tables
|
2. Run bootstrap scripts to initiate the database and tables
|
||||||
|
|
||||||
```text
|
```text
|
||||||
cd $METADATA_HOME
|
cd openmetadata-0.3.0
|
||||||
./boostrap/bootstrap-storage.sh migrate
|
./boostrap/bootstrap-storage.sh migrate
|
||||||
```
|
```
|
||||||
|
|
||||||
3. Start the OpenMetadata Server
|
3. Start the OpenMetadata Server
|
||||||
|
|
||||||
```text
|
```text
|
||||||
|
cd openmetadata-0.3.0
|
||||||
./bin/openmetadata.sh start
|
./bin/openmetadata.sh start
|
||||||
```
|
```
|
||||||
|
|
||||||
Loading…
x
Reference in New Issue
Block a user