datahub/metadata-ingestion/scripts/codegen.sh

#!/bin/bash
set -euo pipefail

OUTDIR=./src/datahub/metadata

# Note: this assumes that datahub has already been built with `./gradlew build`.
DATAHUB_ROOT=..
SCHEMAS_ROOT="$DATAHUB_ROOT/metadata-events/mxe-schemas/src/renamed/avro/com/linkedin"

rm -r $OUTDIR 2>/dev/null || true
python scripts/avro_codegen.py $SCHEMAS_ROOT $OUTDIR
Codegen avro + datahub kafka sink (#3) * Add codegen * New architecture + setup file -> console pipeline * Cleanup source loader * Basic Kafka metadata source * Kafka source and extractor * Add kwargs construct interface * Fix kafka source unit test * start working on pipeline test * kafka datahub sink * Make myself a profile * Ingest to datahub from kafka * Update codegen * Add restli transport * Fix bug in restli conversion 2021-02-05 21:03:04 -08:00			`#!/bin/bash`
fix(ingest): switch to avro from deprecated avro-python3 (#3412) 2021-10-18 18:35:27 -04:00			`set -euo pipefail`
Codegen avro + datahub kafka sink (#3) * Add codegen * New architecture + setup file -> console pipeline * Cleanup source loader * Basic Kafka metadata source * Kafka source and extractor * Add kwargs construct interface * Fix kafka source unit test * start working on pipeline test * kafka datahub sink * Make myself a profile * Ingest to datahub from kafka * Update codegen * Add restli transport * Fix bug in restli conversion 2021-02-05 21:03:04 -08:00
gometa -> datahub 2021-02-15 15:04:21 -08:00			`OUTDIR=./src/datahub/metadata`
Codegen avro + datahub kafka sink (#3) * Add codegen * New architecture + setup file -> console pipeline * Cleanup source loader * Basic Kafka metadata source * Kafka source and extractor * Add kwargs construct interface * Fix kafka source unit test * start working on pipeline test * kafka datahub sink * Make myself a profile * Ingest to datahub from kafka * Update codegen * Add restli transport * Fix bug in restli conversion 2021-02-05 21:03:04 -08:00
			# Note: this assumes that datahub has already been built with `./gradlew build`.
Remove codegen files 2021-02-15 11:44:29 -08:00			`DATAHUB_ROOT=..`
feat: usage stats (part 1) (#2750) Co-authored-by: Gabe Lyons <itsgabelyons@gmail.com> 2021-06-24 17:11:00 -07:00			`SCHEMAS_ROOT="$DATAHUB_ROOT/metadata-events/mxe-schemas/src/renamed/avro/com/linkedin"`
feat(build): adding support for python codegen for all aspects, not just the snapshot ones (#3299) 2021-09-26 17:22:58 -07:00
feat(ingest): infer aspectName from aspect type in MCP (#5566) 2022-08-07 14:52:58 +00:00			`rm -r $OUTDIR 2>/dev/null \|\| true`
			`python scripts/avro_codegen.py $SCHEMAS_ROOT $OUTDIR`