9 Commits

Author SHA1 Message Date
Mayuri Nehate
529710ab9d
fix(ingest/databricks): handle and report config parse failure, updat… (#10261) 2024-04-17 12:14:16 -07:00
Mayuri Nehate
f2e78db92e
feat(ingest/databricks): ingest hive metastore by default, more docs (#9601)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2024-01-18 14:26:33 -08:00
Mayuri Nehate
aac1c55a14
feat(ingest/unity): enable hive metastore ingestion (#9416) 2023-12-14 10:35:06 -05:00
Andrew Sikowitz
27f23ecdd5
feat(ingest/unity): GE Profiling (#8951) 2023-12-06 13:59:23 -05:00
Andrew Sikowitz
66806a805e
feat(ingest/unity): Set external url for containers and datasets (#8238) 2023-06-15 09:05:49 +02:00
Andrew Sikowitz
d3cd4dbb0c
feat(ingest/unity): Allow ingestion without metastore admin role (#8091)
- Adds more detailed docs and connection test
- Fixes empty username queries
2023-05-24 15:36:22 -07:00
Andrew Sikowitz
afcf462cb1
feat(ingest/unity): Add profiling support (#7976)
- Also adds a new databricks sdk
2023-05-11 10:00:50 -07:00
Mayuri Nehate
0131aeefb1
fix(ingest/unity): improve error message if no scheme in workspace_url (#7951)
Co-authored-by: John Joyce <john@acryl.io>
2023-05-08 10:13:53 -07:00
Andrew Sikowitz
5b290c9bc5
feat(ingest/unity): Add usage extraction; add TableReference (#7910)
- Adds usage extraction to the unity catalog source and a TableReference object to handle references to tables
Also makes the following refactors:
- Creates UsageAggregator class to usage_common, as I've seen this same logic multiple times.
- Allows customizable user_urn_builder in usage_common as not all unity users are emails. We create emails with a default email_domain config in other connectors like redshift and snowflake, which seems unnecessary now?
- Creates TableReference for unity catalog and adds it to the Table dataclass, for managing string references to tables. Replaces logic, especially in lineage extraction, with these references
- Creates gen_dataset_urn and gen_user_urn on unity source to reduce duplicate code
Breaks up proxy.py into implementation and types
2023-05-01 11:30:09 -07:00