Ravindra Lanka 108b492ed1
feat(ingestion): Add Iceberg source (#5010)
Co-authored-by: cccs-eric <eric.ladouceur@cyber.gc.ca>
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
2022-05-26 08:05:57 -07:00

2.0 KiB

Concept Mapping

This ingestion source maps the following Source System Concepts to DataHub Concepts:

Source Concept DataHub Concept Notes
iceberg Data Platform
Table Dataset Each Iceberg table maps to a Dataset named using the parent folders. If a table is stored under my/namespace/table, the dataset name will be my.namespace.table. If a Platform Instance is configured, it will be used as a prefix: <platform_instance>.my.namespace.table.
Table property User (a.k.a CorpUser) The value of a table property can be used as the name of a CorpUser owner. This table property name can be configured with the source option user_ownership_property.
Table property CorpGroup The value of a table property can be used as the name of a CorpGroup owner. This table property name can be configured with the source option group_ownership_property.
Table parent folders (excluding warehouse catalog location) Container Available in a future release
Table schema SchemaField Maps to the fields defined within the Iceberg table schema definition.

Troubleshooting

[Common Issue]

[Provide description of common issues with this integration and steps to resolve]