6 Commits

Author	SHA1	Message	Date
Andrew Sikowitz	5b290c9bc5	feat(ingest/unity): Add usage extraction; add TableReference (#7910) ... - Adds usage extraction to the unity catalog source and a TableReference object to handle references to tables Also makes the following refactors: - Creates UsageAggregator class to usage_common, as I've seen this same logic multiple times. - Allows customizable user_urn_builder in usage_common as not all unity users are emails. We create emails with a default email_domain config in other connectors like redshift and snowflake, which seems unnecessary now? - Creates TableReference for unity catalog and adds it to the Table dataclass, for managing string references to tables. Replaces logic, especially in lineage extraction, with these references - Creates gen_dataset_urn and gen_user_urn on unity source to reduce duplicate code Breaks up proxy.py into implementation and types	2023-05-01 11:30:09 -07:00
Andrew Sikowitz	e9c2f9afcc	feat(ingest/unity): Ingest ownership for containers; lookup service principal display names (#7869)	2023-04-21 11:02:39 -07:00
Harshal Sheth	e461d03d94	feat(ingest/unity): capture create/lastModified timestamps (#7819) ... Co-authored-by: Tamas Nemeth <treff7es@gmail.com>	2023-04-17 12:18:21 -07:00
Harshal Sheth	204727a6ee	feat(ingest/unity): support extracting ownership (#7801)	2023-04-12 19:45:41 -07:00
Tamas Nemeth	62e33e03a3	fix(ingest/unity): Use assigned metastore if not metastore listed in unity catalog (#7446)	2023-02-28 08:06:28 +01:00
Tamas Nemeth	0dc2d6af25	feat(ingest): databricks - Unity catalog source (#6292) ... Co-authored-by: MohdSiddique Bagwan <mohdsiddique.bagwan@gslab.com> Co-authored-by: Shirshanka Das <shirshanka@apache.org>	2022-10-29 21:44:41 -07:00