yujunjun/datahub
2
0
Fork 0
mirror of https://github.com/datahub-project/datahub.git synced 2025-08-15 12:46:53 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs(metadata-models-custom): add example script to show producing cu… (#4681)

Browse Source
This commit is contained in:
Shirshanka Das 2022-04-18 17:08:06 -07:00 committed by GitHub
parent 9ecb785698
commit c5b8f57fb3
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
2 changed files with 57 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

9
metadata-models-custom/README.md
View File

@ -129,6 +129,15 @@ results in
Update succeeded with status 200 Update succeeded with status 200
``` ```
The `scripts/insert_custom_aspect.py` script shows you how to accomplish the same using the Python SDK. Note that we are just using a raw dictionary here to represent the `dq_rule` aspect and not a strongly-typed class.
```console
cd scripts
python3 insert_custom_aspect.py
```
results in
```console
Successfully wrote to DataHub
```
## Advanced Guide ## Advanced Guide

48
metadata-models-custom/scripts/insert_custom_aspect.py Normal file
View File

@ -0,0 +1,48 @@
import json
from datahub.emitter.rest_emitter import DatahubRestEmitter
from datahub.metadata.schema_classes import (
ChangeTypeClass,
GenericAspectClass,
MetadataChangeProposalClass,
)
dq_aspect = {
"rules": [
{
"field": "my_event_data",
"isFieldLevel": False,
"type": "isNull",
"checkDefinition": "n/a",
"url": "https://github.com/datahub-project/datahub/blob/master/checks/nonNull.sql",
},
{
"field": "timestamp",
"isFieldLevel": True,
"type": "increasing",
"checkDefinition": "n/a",
"url": "https://github.com/datahub-project/datahub/blob/master/checks/increasing.sql",
},
]
}
emitter: DatahubRestEmitter = DatahubRestEmitter(gms_server="http://localhost:8080")
dataset_urn = "urn:li:dataset:(urn:li:dataPlatform:hive,logging_events,PROD)"
mcp_raw: MetadataChangeProposalClass = MetadataChangeProposalClass(
entityType="dataset",
entityUrn=dataset_urn,
changeType=ChangeTypeClass.UPSERT,
aspectName="customDataQualityRules",
aspect=GenericAspectClass(
contentType="application/json",
value=json.dumps(dq_aspect).encode("utf-8"),
),
)
try:
emitter.emit(mcp_raw)
print("Successfully wrote to DataHub")
except Exception as e:
print("Failed to write to DataHub")
raise e