datahub/metadata-ingestion/scripts/codegen.sh

#!/bin/bash
set -euxo pipefail

OUTDIR=./src/datahub/metadata

# Note: this assumes that datahub has already been built with `./gradlew build`.
DATAHUB_ROOT=..
cp $DATAHUB_ROOT/metadata-events/mxe-schemas/src/renamed/avro/com/linkedin/mxe/MetadataChangeEvent.avsc .

rm -r $OUTDIR || true
python scripts/avro_codegen.py MetadataChangeEvent.avsc $OUTDIR
rm MetadataChangeEvent.avsc
Codegen avro + datahub kafka sink (#3) * Add codegen * New architecture + setup file -> console pipeline * Cleanup source loader * Basic Kafka metadata source * Kafka source and extractor * Add kwargs construct interface * Fix kafka source unit test * start working on pipeline test * kafka datahub sink * Make myself a profile * Ingest to datahub from kafka * Update codegen * Add restli transport * Fix bug in restli conversion 2021-02-05 21:03:04 -08:00			`#!/bin/bash`
			`set -euxo pipefail`

gometa -> datahub 2021-02-15 15:04:21 -08:00			`OUTDIR=./src/datahub/metadata`
Codegen avro + datahub kafka sink (#3) * Add codegen * New architecture + setup file -> console pipeline * Cleanup source loader * Basic Kafka metadata source * Kafka source and extractor * Add kwargs construct interface * Fix kafka source unit test * start working on pipeline test * kafka datahub sink * Make myself a profile * Ingest to datahub from kafka * Update codegen * Add restli transport * Fix bug in restli conversion 2021-02-05 21:03:04 -08:00
			# Note: this assumes that datahub has already been built with `./gradlew build`.
Remove codegen files 2021-02-15 11:44:29 -08:00			`DATAHUB_ROOT=..`
Codegen avro + datahub kafka sink (#3) * Add codegen * New architecture + setup file -> console pipeline * Cleanup source loader * Basic Kafka metadata source * Kafka source and extractor * Add kwargs construct interface * Fix kafka source unit test * start working on pipeline test * kafka datahub sink * Make myself a profile * Ingest to datahub from kafka * Update codegen * Add restli transport * Fix bug in restli conversion 2021-02-05 21:03:04 -08:00			`cp $DATAHUB_ROOT/metadata-events/mxe-schemas/src/renamed/avro/com/linkedin/mxe/MetadataChangeEvent.avsc .`

			`rm -r $OUTDIR \|\| true`
			`python scripts/avro_codegen.py MetadataChangeEvent.avsc $OUTDIR`
			`rm MetadataChangeEvent.avsc`