mirror of https://github.com/datahub-project/datahub.git synced 2025-12-11 01:56:58 +00:00

History

fix: ingestion docker image (#2027 )

The environment was not set correctly, so it could not fire kafka events. It (mce-cli) always worked when running outside of docker.

I also added a dev ingestion docker image / script which may be much faster if you've already built locally.

Tested:
1. Cleaned docker volumes and started datahub. Verified it is empty.
2. Built with gradle.
3. Ran ./docker/ingestion/ingestion-dev.sh. Verified data shows in DataHub.
4. Ran step 1 again.
5. Ran ./docker/ingestion/ingestion.sh. Verified data shows in DataHub.

2020-12-02 17:40:12 -08:00

src

feat: Port mce-cli to Java. (#1871 )

2020-09-25 14:05:29 -07:00

build.gradle

feat: Port mce-cli to Java. (#1871 )

2020-09-25 14:05:29 -07:00

example-bootstrap.json

feat: Port mce-cli to Java. (#1871 )

2020-09-25 14:05:29 -07:00

README.md

fix: ingestion docker image (#2027 )

2020-12-02 17:40:12 -08:00

README.md

MCE CLI

A small application which can produce or consume MetadataChangeEvents.

Running the Application

First, ensure that services this depends on, like schema registry / zookeeper / mce-consumer-job / gms / etc, are all running.

This application can be run via gradle:

./gradlew :metadata-ingestion-examples:mce-cli:bootRun

Or by building and running the jar:

./gradlew :metadata-ingestion-examples:mce-cli:build

java -jar metadata-ingestion-examples/mce-cli/build/libs/mce-cli.jar

Consuming Events

Consuming MCEs may be useful to help debug other applications that are meant to produce them. You can easily see what MCEs are being produced (or not) at a glance.

./gradlew :metadata-ingestion-examples:mce-cli:bootRun

# Alternatives
./gradlew :metadata-ingestion-examples:mce-cli:bootRun --args='consume'
java -jar  metadata-ingestion-examples/mce-cli/build/libs/mce-cli.jar
java -jar metadata-ingestion-examples/mce-cli/build/libs/mce-cli.jar consume

Producing Events

Producing events can be useful to help debug the MCE pipeline, or just to help make some fake data (ideally, don't do this on your production stack!).

./gradlew :metadata-ingestion-examples:mce-cli:bootRun --args='-m produce my-file.json'

# Alternatively
java -jar  metadata-ingestion-examples/mce-cli/build/libs/mce-cli.jar -m produce my-file.json

Where my-file.json is some file that contains a MetadataChangEvents JSON object.

Producing the Example Events with Docker

We have some example events in the example-bootstrap.json file, which can be invoked via the above example or in a docker environment using docker/ingestion/ingestion.sh. We also have a developer image (docker/ingestion/ingestion-dev.sh) which uses your locally built jar rather than building on the docker image itself, which may be faster if you have already built code locally.