yujunjun/datahub
2
0
Fork 0
mirror of https://github.com/datahub-project/datahub.git synced 2025-11-30 04:34:34 +00:00
Code Issues Packages Projects Releases Wiki Activity
datahub/metadata-jobs/elasticsearch-index-job
History
Kerem Sahin 8e809e383d Fix Kafka topic names
2019-10-02 16:28:03 -07:00
..
src/main
Fix Kafka topic names
2019-10-02 16:28:03 -07:00
build.gradle
Fix Kafka topic names
2019-10-02 16:28:03 -07:00
README.md
Update documentation
2019-09-09 01:45:49 -07:00

README.md

MetadataAuditEvent (MAE) Consumer Job

MAE Consumer is a Kafka Streams job. Its main function is to listen MetadataAuditEvent Kafka topic for messages and process those messages using index builders. Index builders create search document model by processing MAE and then these documents are indexed into Elasticsearch. So, this job is providing us a near-realtime search index update.

Pre-requisites

  • You need to have JDK8 installed on your machine to be able to build Data Hub GMS.

Build

MAE Consumer Job is already built as part of top level build:

./gradlew build

However, if you only want to build MAE Consumer Job specifically:

./gradlew :metadata-jobs:elasticsearch-index-job:build

Dependencies

Before starting MAE Consumer Job, you need to make sure that Kafka, Schema Registry & Zookeeper and
Elasticsearch Docker containers are up and running.

Start via Docker image

Quickest way to try out MAE Consumer Job is running the Docker image.

Start via command line

If you do modify things and want to try it out quickly without building the Docker image, you can also run the application directly from command line after a successful build:

./gradlew :metadata-jobs:elasticsearch-index-job:run