datahub/metadata-ingestion/scripts/codegen.sh

#!/bin/bash
set -euxo pipefail

OUTDIR=./src/datahub/metadata

# Note: this assumes that datahub has already been built with `./gradlew build`.
DATAHUB_ROOT=..
SCHEMAS_ROOT="$DATAHUB_ROOT/metadata-events/mxe-schemas/src/renamed/avro/com/linkedin/mxe"
FILES="$SCHEMAS_ROOT/MetadataChangeEvent.avsc $SCHEMAS_ROOT/MetadataAuditEvent.avsc"

rm -r $OUTDIR || true
python scripts/avro_codegen.py $FILES $OUTDIR
Codegen avro + datahub kafka sink (#3) * Add codegen * New architecture + setup file -> console pipeline * Cleanup source loader * Basic Kafka metadata source * Kafka source and extractor * Add kwargs construct interface * Fix kafka source unit test * start working on pipeline test * kafka datahub sink * Make myself a profile * Ingest to datahub from kafka * Update codegen * Add restli transport * Fix bug in restli conversion 2021-02-05 21:03:04 -08:00			`#!/bin/bash`
			`set -euxo pipefail`

gometa -> datahub 2021-02-15 15:04:21 -08:00			`OUTDIR=./src/datahub/metadata`
Codegen avro + datahub kafka sink (#3) * Add codegen * New architecture + setup file -> console pipeline * Cleanup source loader * Basic Kafka metadata source * Kafka source and extractor * Add kwargs construct interface * Fix kafka source unit test * start working on pipeline test * kafka datahub sink * Make myself a profile * Ingest to datahub from kafka * Update codegen * Add restli transport * Fix bug in restli conversion 2021-02-05 21:03:04 -08:00
			# Note: this assumes that datahub has already been built with `./gradlew build`.
Remove codegen files 2021-02-15 11:44:29 -08:00			`DATAHUB_ROOT=..`
feat(ingest): expose additional types to Python via codegen (#2712) 2021-06-17 10:04:28 -07:00			`SCHEMAS_ROOT="$DATAHUB_ROOT/metadata-events/mxe-schemas/src/renamed/avro/com/linkedin/mxe"`
			`FILES="$SCHEMAS_ROOT/MetadataChangeEvent.avsc $SCHEMAS_ROOT/MetadataAuditEvent.avsc"`
Codegen avro + datahub kafka sink (#3) * Add codegen * New architecture + setup file -> console pipeline * Cleanup source loader * Basic Kafka metadata source * Kafka source and extractor * Add kwargs construct interface * Fix kafka source unit test * start working on pipeline test * kafka datahub sink * Make myself a profile * Ingest to datahub from kafka * Update codegen * Add restli transport * Fix bug in restli conversion 2021-02-05 21:03:04 -08:00
			`rm -r $OUTDIR \|\| true`
feat(ingest): expose additional types to Python via codegen (#2712) 2021-06-17 10:04:28 -07:00			`python scripts/avro_codegen.py $FILES $OUTDIR`