yujunjun/OpenMetadata
1
0
Fork 0
mirror of https://github.com/open-metadata/OpenMetadata.git synced 2025-07-22 17:01:41 +00:00
Code Issues Packages Projects Releases Wiki Activity
OpenMetadata/ingestion
History
parthp2107 06810cdec1
Fix #432:Added Redash Connector (#444) 
* added redash connector

* added redash connector

* Added Redash Connector

* minor changes

Co-authored-by: parthp2107 <parth.panchal@deuexsoultions.com>
Co-authored-by: parthp2107 <parth@getcollate.io>
2021-09-22 15:09:24 +05:30
..
examples
Fix #432:Added Redash Connector (#444)
2021-09-22 15:09:24 +05:30
ingestion_scheduler
Few Pipeline fixes and Documentation updated
2021-08-24 18:49:52 +05:30
pipelines
Setup.py Refactored, ES port fix (#521)
2021-09-19 13:59:14 +05:30
src/metadata
Fix #432:Added Redash Connector (#444)
2021-09-22 15:09:24 +05:30
tests
added unit tests to helper and workflow classes in ingestion (#391)
2021-09-12 01:07:13 +05:30
CHANGELOG
OpenMetadata snapshot release 0.3
2021-08-01 14:27:44 -07:00
Dockerfile
Docker fix (#200)
2021-08-16 18:37:04 +05:30
ingestion_dependency.sh
Pip install git path modified pylint - simplescheduler
2021-08-18 01:01:20 +05:30
LICENSE
Ingestion Setup.py modification
2021-08-13 01:40:56 +05:30
README.md
README and Setup.py Modification
2021-08-13 02:03:39 +05:30
requirements.txt
[WIP] profiler
2021-09-07 22:03:57 -07:00
setup.cfg
Ingestion Setup.py modification
2021-08-13 01:40:56 +05:30
setup.py
Fix #432:Added Redash Connector (#444)
2021-09-22 15:09:24 +05:30

README.md

This guide will help you setup the Ingestion framework and connectors
This guide will help you setup the Ingestion framework and connectors

Python version 3.8+

OpenMetadata Ingesiton is a simple framework to build connectors and ingest metadata of various systems through OpenMetadata APIs. It could be used in an orchestration framework(e.g. Apache Airflow) to ingest metadata. Prerequisites

  • Python >= 3.8.x

Install From PyPI

python3 -m pip install --upgrade pip wheel setuptools openmetadata-ingestion
python3 -m spacy download en_core_web_sm

Install Ingestion Connector Dependencies

Click here to go to Ingestion Connector's Documentation

Generate Redshift Data

metadata ingest -c ./pipelines/redshift.json

Generate Redshift Usage Data

metadata ingest -c ./pipelines/redshift_usage.json

Generate Sample Tables

metadata ingest -c ./pipelines/sample_tables.json

Generate Sample Users

metadata ingest -c ./pipelines/sample_users.json

Ingest MySQL data to Metadata APIs

metadata ingest -c ./pipelines/mysql.json

Ingest Bigquery data to Metadata APIs

export GOOGLE_APPLICATION_CREDENTIALS="$PWD/pipelines/creds/bigquery-cred.json"
metadata ingest -c ./pipelines/bigquery.json

Index Metadata into ElasticSearch

Run ElasticSearch docker

docker run -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" docker.elastic.co/elasticsearch/elasticsearch:7.10.2

Run ingestion connector

metadata ingest -c ./pipelines/metadata_to_es.json