From 40911af11f365c48fd60a5f333fe603c347e056d Mon Sep 17 00:00:00 2001 From: Suresh Srinivas Date: Thu, 12 Aug 2021 11:39:56 -0700 Subject: [PATCH] Update setup, ingestion docs --- docs/SUMMARY.md | 3 +- docs/install/metadata-ingestion/README.md | 19 +++++++ docs/install/run-in-production.md | 56 +++++++++++++++++++ .../{run-locally.md => run-openmetadata.md} | 46 +++++++++++---- 4 files changed, 111 insertions(+), 13 deletions(-) rename docs/install/{run-locally.md => run-openmetadata.md} (76%) diff --git a/docs/SUMMARY.md b/docs/SUMMARY.md index 21d04efecc9..5ebe3a62153 100644 --- a/docs/SUMMARY.md +++ b/docs/SUMMARY.md @@ -37,8 +37,7 @@ ## Install -* [Run Locally](install/run-locally.md) -* [Run Docker](install/run-docker.md) +* [Run OpenMetadata](install/run-openmetadata.md) * [Run in production](install/run-in-production.md) * [Metadata Ingestion](install/metadata-ingestion/README.md) * [Ingest Sample Data](install/metadata-ingestion/ingest-sample-data.md) diff --git a/docs/install/metadata-ingestion/README.md b/docs/install/metadata-ingestion/README.md index c20d238788d..a5c7d45c07a 100644 --- a/docs/install/metadata-ingestion/README.md +++ b/docs/install/metadata-ingestion/README.md @@ -1,2 +1,21 @@ # Metadata Ingestion +OpenMetadata Ingesiton is a simple framework to build connectors and ingest metadata of various systems through OpenMetadata APIs. +It could be used in an orchestration framework(e.g. Apache Airflow) to ingest metadata. + +## Ingest Sample Data +{% page-ref page="ingest-sample-metadata" %} + + +## Explore and configure connectors +{% page-ref page="connectors" %} + + +## Run Simple Scheduler + +{% page-ref page="scheduler" %} + + +## Run using Airflow + +{% page-ref page="airflow" %} diff --git a/docs/install/run-in-production.md b/docs/install/run-in-production.md index 481902e9181..667c06ad875 100644 --- a/docs/install/run-in-production.md +++ b/docs/install/run-in-production.md @@ -1,2 +1,58 @@ +--- +description: >- + This installation doc will help you start a OpenMetadata instances + on your production. +--- + + # Run in production +Please refer to the previous section [Run Openmetadata](install/run-openmetadata.md) for configuring OpenMetadata. + +{% hint style="info" %} +**Prerequisites** + +* MySQL >= 8.x +* ElasticSearch >= 7.x +* Airflow or other schedulers to run Ingestion Connectors + +{% endhint %} + + +## Start OpenMetadata + +OpenMetadata release ships with ```./bin/openmetadata``` init.d style +script. + +```text +cd openmetdata-0.3.0 +./bin/openmetdata start +``` + +We recommend to configure serviced to monitor openmetadata command to restart +incase of any failures. + + +## Running with a load balancer + +One or more OpenMetadata instances can be put behind a load balancer for reverse proxying, +in that case appropriate OpenMetdata url must be mentioned in the load balancer's configuraiton file. + +For example, in case Apache mod proxy the VirtualHost tag in the configuration file should be +edited out with the following + +```text + + + + BalancerMember http://127.0.0.1:8585 + BalancerMember http://127.0.0.2:8686 + + + ProxyPreserveHost On + + ProxyPass / balancer://mycluster/ + ProxyPassReverse / balancer://mycluster/ + + ``` + \ No newline at end of file diff --git a/docs/install/run-locally.md b/docs/install/run-openmetadata.md similarity index 76% rename from docs/install/run-locally.md rename to docs/install/run-openmetadata.md index 73205f40e23..2b30f933847 100644 --- a/docs/install/run-locally.md +++ b/docs/install/run-openmetadata.md @@ -4,6 +4,27 @@ description: >- on your local machine. --- +# Run Docker + +[Docker](https://docs.docker.com/get-started/overview/) is an open platform for developing, shipping, and running applications that enables you to separate your applications from your infrastructure so you can deliver software quickly using OS-level virtualization to deliver software in packages called containers. + +{% hint style="info" %} +**Prerequisites** + +* Docker >= 20.10.x + +**Ports to access once the docker is up:** + +* OpenMetadata UI: 8585 +* Scheduler UI: 7777 +{% endhint %} + +```text +cd docker/metadata +docker-compose up +``` + + # Run Locally {% hint style="success" %} @@ -22,6 +43,18 @@ OpenMetadata is built using Java, DropWizard, Jetty, and MySQL. {% endhint %} {% tabs %} +{% tab title="Download the release" %} +Download the latest binary release from [OpenMetadata](https://open-metadata.org/download/), Once you have the tar file, + +```bash +# untar it +tar -zxvf openmetadata-0.3.0.tar.gz + +# navigate to directory containing the launcher scripts +cd openmetadata-0.3.0 +``` +{% endtab %} + {% tab title="Build from source " %} Follow these steps to checkout code from [Github](https://github.com/open-metadata/OpenMetadata) and build OpenMetadata locally @@ -46,17 +79,7 @@ cd openmetadata-1.0.0-SNAPSHOT ``` {% endtab %} -{% tab title="Download the release" %} -Download the latest binary release from [OpenMetadata](https://open-metadata.org/download/), Once you have the tar file, -```bash -# untar it -tar -zxvf openmetadata-1.0.0-SNAPSHOT.tar.gz - -# navigate to directory containing the launcher scripts -cd openmetadata-1.0.0-SNAPSHOT -``` -{% endtab %} {% endtabs %} ## Install on your local machine @@ -91,13 +114,14 @@ cd openmetadata-1.0.0-SNAPSHOT 2. Run bootstrap scripts to initiate the database and tables ```text - cd $METADATA_HOME + cd openmetadata-0.3.0 ./boostrap/bootstrap-storage.sh migrate ``` 3. Start the OpenMetadata Server ```text + cd openmetadata-0.3.0 ./bin/openmetadata.sh start ```