datahub/metadata-ingestion/scripts/codegen.sh

#!/bin/bash
set -euxo pipefail

OUTDIR=./src/datahub/metadata

# Note: this assumes that datahub has already been built with `./gradlew build`.
DATAHUB_ROOT=..
SCHEMAS_ROOT="$DATAHUB_ROOT/metadata-events/mxe-schemas/src/renamed/avro/com/linkedin"
FILES="$SCHEMAS_ROOT/mxe/MetadataChangeEvent.avsc $SCHEMAS_ROOT/mxe/MetadataChangeProposal.avsc $SCHEMAS_ROOT/usage/UsageAggregation.avsc $SCHEMAS_ROOT/dataset/DatasetProfile.avsc"

rm -r $OUTDIR || true
python scripts/avro_codegen.py $FILES $OUTDIR
Codegen avro + datahub kafka sink (#3) * Add codegen * New architecture + setup file -> console pipeline * Cleanup source loader * Basic Kafka metadata source * Kafka source and extractor * Add kwargs construct interface * Fix kafka source unit test * start working on pipeline test * kafka datahub sink * Make myself a profile * Ingest to datahub from kafka * Update codegen * Add restli transport * Fix bug in restli conversion 2021-02-05 21:03:04 -08:00			`#!/bin/bash`
			`set -euxo pipefail`

gometa -> datahub 2021-02-15 15:04:21 -08:00			`OUTDIR=./src/datahub/metadata`
Codegen avro + datahub kafka sink (#3) * Add codegen * New architecture + setup file -> console pipeline * Cleanup source loader * Basic Kafka metadata source * Kafka source and extractor * Add kwargs construct interface * Fix kafka source unit test * start working on pipeline test * kafka datahub sink * Make myself a profile * Ingest to datahub from kafka * Update codegen * Add restli transport * Fix bug in restli conversion 2021-02-05 21:03:04 -08:00
			# Note: this assumes that datahub has already been built with `./gradlew build`.
Remove codegen files 2021-02-15 11:44:29 -08:00			`DATAHUB_ROOT=..`
feat: usage stats (part 1) (#2750) Co-authored-by: Gabe Lyons <itsgabelyons@gmail.com> 2021-06-24 17:11:00 -07:00			`SCHEMAS_ROOT="$DATAHUB_ROOT/metadata-events/mxe-schemas/src/renamed/avro/com/linkedin"`
Introducing TimeSeries Aspects + Dataset Profile (Stats) Aspect (#2983) Co-authored-by: Dexter Lee <dexter@acryl.io> Co-authored-by: Harshal Sheth <hsheth2@gmail.com> Co-authored-by: Ravindra Lanka <rlanka@acryl.io> Co-authored-by: Shirshanka Das <shirshanka@apache.org> 2021-07-30 17:41:03 -07:00			`FILES="$SCHEMAS_ROOT/mxe/MetadataChangeEvent.avsc $SCHEMAS_ROOT/mxe/MetadataChangeProposal.avsc $SCHEMAS_ROOT/usage/UsageAggregation.avsc $SCHEMAS_ROOT/dataset/DatasetProfile.avsc"`
Codegen avro + datahub kafka sink (#3) * Add codegen * New architecture + setup file -> console pipeline * Cleanup source loader * Basic Kafka metadata source * Kafka source and extractor * Add kwargs construct interface * Fix kafka source unit test * start working on pipeline test * kafka datahub sink * Make myself a profile * Ingest to datahub from kafka * Update codegen * Add restli transport * Fix bug in restli conversion 2021-02-05 21:03:04 -08:00
			`rm -r $OUTDIR \|\| true`
feat(ingest): expose additional types to Python via codegen (#2712) 2021-06-17 10:04:28 -07:00			`python scripts/avro_codegen.py $FILES $OUTDIR`