yujunjun/datahub
2
0
Fork 0
mirror of https://github.com/datahub-project/datahub.git synced 2025-07-23 17:39:59 +00:00
Code Issues Packages Projects Releases Wiki Activity

fix(ingest/redshift): include table type in custom properties (#9576)

Browse Source
This commit is contained in:
Harshal Sheth 2024-01-09 16:08:31 -05:00 committed by GitHub
parent e3f2c52d7e
commit cca1e9dd49
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 5 additions and 2 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

7
metadata-ingestion/src/datahub/ingestion/source/redshift/redshift.py
View File

@ -165,7 +165,7 @@ class RedshiftSource(StatefulIngestionSourceBase, TestableSource):
#### sql_based
The sql_based based collector uses Redshift's [stl_insert](https://docs.aws.amazon.com/redshift/latest/dg/r_STL_INSERT.html) to discover all the insert queries
and uses sql parsing to discover the dependecies.
and uses sql parsing to discover the dependencies.
Pros:
- Works with Spectrum tables
@ -189,7 +189,7 @@ class RedshiftSource(StatefulIngestionSourceBase, TestableSource):
:::note
The redshift stl redshift tables which are used for getting data lineage only retain approximately two to five days of log history. This means you cannot extract lineage from queries issued outside that window.
The redshift stl redshift tables which are used for getting data lineage retain at most seven days of log history, and sometimes closer to 2-5 days. This means you cannot extract lineage from queries issued outside that window.
:::
@ -592,6 +592,9 @@ class RedshiftSource(StatefulIngestionSourceBase, TestableSource):
) -> Iterable[MetadataWorkUnit]:
custom_properties = {}
if table.type:
custom_properties["table_type"] = table.type
if table.location:
custom_properties["location"] = table.location