yujunjun/OpenMetadata
1
0
Fork 0
mirror of https://github.com/open-metadata/OpenMetadata.git synced 2025-07-24 17:59:52 +00:00
Code Issues Packages Projects Releases Wiki Activity
OpenMetadata/ingestion
History
parthp2107 d3d0ac19d0
Removed serviceType as a configuration option from workflow configs (#371) 
* Removed serviceType as a configuration option from workflow configs

* Removed serviceType as a configuration option from workflow configs

Co-authored-by: parthp2107 <parth.panchal@deuexsoultions.com>
2021-09-02 19:45:41 +05:30
..
examples
Removed serviceType as a configuration option from workflow configs (#371)
2021-09-02 19:45:41 +05:30
ingestion_scheduler
Few Pipeline fixes and Documentation updated
2021-08-24 18:49:52 +05:30
pipelines
Removed serviceType as a configuration option from workflow configs (#371)
2021-09-02 19:45:41 +05:30
src/metadata
Removed serviceType as a configuration option from workflow configs (#371)
2021-09-02 19:45:41 +05:30
tests
Ingestion Setup.py modification
2021-08-13 01:40:56 +05:30
CHANGELOG
OpenMetadata snapshot release 0.3
2021-08-01 14:27:44 -07:00
Dockerfile
Docker fix (#200)
2021-08-16 18:37:04 +05:30
ingestion_dependency.sh
Pip install git path modified pylint - simplescheduler
2021-08-18 01:01:20 +05:30
LICENSE
Ingestion Setup.py modification
2021-08-13 01:40:56 +05:30
README.md
README and Setup.py Modification
2021-08-13 02:03:39 +05:30
requirements.txt
[WIP] Issue #285: Add support for Dashboard Entities; Superset connector
2021-08-24 13:47:41 -07:00
setup.cfg
Ingestion Setup.py modification
2021-08-13 01:40:56 +05:30
setup.py
[WIP] Issue #285: Add support for Dashboard Entities; Superset connector
2021-08-24 13:47:41 -07:00

README.md

This guide will help you setup the Ingestion framework and connectors
This guide will help you setup the Ingestion framework and connectors

Python version 3.8+

OpenMetadata Ingesiton is a simple framework to build connectors and ingest metadata of various systems through OpenMetadata APIs. It could be used in an orchestration framework(e.g. Apache Airflow) to ingest metadata. Prerequisites

  • Python >= 3.8.x

Install From PyPI

python3 -m pip install --upgrade pip wheel setuptools openmetadata-ingestion
python3 -m spacy download en_core_web_sm

Install Ingestion Connector Dependencies

Click here to go to Ingestion Connector's Documentation

Generate Redshift Data

metadata ingest -c ./pipelines/redshift.json

Generate Redshift Usage Data

metadata ingest -c ./pipelines/redshift_usage.json

Generate Sample Tables

metadata ingest -c ./pipelines/sample_tables.json

Generate Sample Users

metadata ingest -c ./pipelines/sample_users.json

Ingest MySQL data to Metadata APIs

metadata ingest -c ./pipelines/mysql.json

Ingest Bigquery data to Metadata APIs

export GOOGLE_APPLICATION_CREDENTIALS="$PWD/pipelines/creds/bigquery-cred.json"
metadata ingest -c ./pipelines/bigquery.json

Index Metadata into ElasticSearch

Run ElasticSearch docker

docker run -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" docker.elastic.co/elasticsearch/elasticsearch:7.10.2

Run ingestion connector

metadata ingest -c ./pipelines/metadata_to_es.json