3 Commits

Author SHA1 Message Date
Andrew Sikowitz
afcf462cb1
feat(ingest/unity): Add profiling support (#7976)
- Also adds a new databricks sdk
2023-05-11 10:00:50 -07:00
Mayuri Nehate
0131aeefb1
fix(ingest/unity): improve error message if no scheme in workspace_url (#7951)
Co-authored-by: John Joyce <john@acryl.io>
2023-05-08 10:13:53 -07:00
Andrew Sikowitz
5b290c9bc5
feat(ingest/unity): Add usage extraction; add TableReference (#7910)
- Adds usage extraction to the unity catalog source and a TableReference object to handle references to tables
Also makes the following refactors:
- Creates UsageAggregator class to usage_common, as I've seen this same logic multiple times.
- Allows customizable user_urn_builder in usage_common as not all unity users are emails. We create emails with a default email_domain config in other connectors like redshift and snowflake, which seems unnecessary now?
- Creates TableReference for unity catalog and adds it to the Table dataclass, for managing string references to tables. Replaces logic, especially in lineage extraction, with these references
- Creates gen_dataset_urn and gen_user_urn on unity source to reduce duplicate code
Breaks up proxy.py into implementation and types
2023-05-01 11:30:09 -07:00