datahub/metadata-ingestion/scripts/codegen.sh

#!/bin/bash
set -euo pipefail

OUTDIR=./src/datahub/metadata

# Note: this assumes that datahub has already been built with `./gradlew build`.
DATAHUB_ROOT=..

SCHEMAS_PDL="$DATAHUB_ROOT/metadata-models/src/main/pegasus/com/linkedin"
SCHEMAS_AVSC="$DATAHUB_ROOT/metadata-events/mxe-schemas/src/renamed/avro/com/linkedin"
ENTITY_REGISTRY="$DATAHUB_ROOT/metadata-models/src/main/resources/entity-registry.yml"

rm -r $OUTDIR 2>/dev/null || true
python scripts/avro_codegen.py $ENTITY_REGISTRY $SCHEMAS_PDL $SCHEMAS_AVSC $OUTDIR
Codegen avro + datahub kafka sink (#3) * Add codegen * New architecture + setup file -> console pipeline * Cleanup source loader * Basic Kafka metadata source * Kafka source and extractor * Add kwargs construct interface * Fix kafka source unit test * start working on pipeline test * kafka datahub sink * Make myself a profile * Ingest to datahub from kafka * Update codegen * Add restli transport * Fix bug in restli conversion 2021-02-05 21:03:04 -08:00			`#!/bin/bash`
fix(ingest): switch to avro from deprecated avro-python3 (#3412) 2021-10-18 18:35:27 -04:00			`set -euo pipefail`
Codegen avro + datahub kafka sink (#3) * Add codegen * New architecture + setup file -> console pipeline * Cleanup source loader * Basic Kafka metadata source * Kafka source and extractor * Add kwargs construct interface * Fix kafka source unit test * start working on pipeline test * kafka datahub sink * Make myself a profile * Ingest to datahub from kafka * Update codegen * Add restli transport * Fix bug in restli conversion 2021-02-05 21:03:04 -08:00
gometa -> datahub 2021-02-15 15:04:21 -08:00			`OUTDIR=./src/datahub/metadata`
Codegen avro + datahub kafka sink (#3) * Add codegen * New architecture + setup file -> console pipeline * Cleanup source loader * Basic Kafka metadata source * Kafka source and extractor * Add kwargs construct interface * Fix kafka source unit test * start working on pipeline test * kafka datahub sink * Make myself a profile * Ingest to datahub from kafka * Update codegen * Add restli transport * Fix bug in restli conversion 2021-02-05 21:03:04 -08:00
			# Note: this assumes that datahub has already been built with `./gradlew build`.
Remove codegen files 2021-02-15 11:44:29 -08:00			`DATAHUB_ROOT=..`
feat(ingest): add urn modification helper (#7440) Co-authored-by: Tamas Nemeth <treff7es@gmail.com> 2023-03-16 16:27:08 -04:00
			`SCHEMAS_PDL="$DATAHUB_ROOT/metadata-models/src/main/pegasus/com/linkedin"`
			`SCHEMAS_AVSC="$DATAHUB_ROOT/metadata-events/mxe-schemas/src/renamed/avro/com/linkedin"`
feat(ingest): add entity registry in codegen (#6984) Co-authored-by: Pedro Silva <pedro@acryl.io> 2023-01-17 19:41:43 -08:00			`ENTITY_REGISTRY="$DATAHUB_ROOT/metadata-models/src/main/resources/entity-registry.yml"`
feat(build): adding support for python codegen for all aspects, not just the snapshot ones (#3299) 2021-09-26 17:22:58 -07:00
feat(ingest): infer aspectName from aspect type in MCP (#5566) 2022-08-07 14:52:58 +00:00			`rm -r $OUTDIR 2>/dev/null \|\| true`
feat(ingest): add urn modification helper (#7440) Co-authored-by: Tamas Nemeth <treff7es@gmail.com> 2023-03-16 16:27:08 -04:00			`python scripts/avro_codegen.py $ENTITY_REGISTRY $SCHEMAS_PDL $SCHEMAS_AVSC $OUTDIR`