Sergio Gómez Villamor
8cae980286
tests(ingestion): moving some tests so they are available for sdk users ( #13540 )
2025-05-19 08:39:53 +02:00
skrydal
65d1c2b43c
feat(ingestion): Make jsonProps of schemaMetadata less verbose ( #13416 )
2025-05-06 16:18:26 +02:00
Chakru
54156ea78a
fix(cli): use patch to update dataset properties ( #13226 )
2025-04-21 14:41:31 -07:00
Aseem Bansal
9c3bd34995
dev: enable ruff rule ( #12749 )
2025-02-28 17:49:52 +05:30
Aseem Bansal
262dd76518
dev: remove black in favor of ruff for formatting ( #12378 )
2025-01-18 15:06:20 +05:30
Harshal Sheth
5519a330e2
chore(ingest): bump black ( #11898 )
2024-11-20 13:33:54 -08:00
Sergio Gómez Villamor
31edb46dbc
feat(ingestion): adds env property in ContainerProperties ( #11214 )
...
Co-authored-by: siladitya2 <siladitya2@gmail.com>
2024-09-18 14:56:52 +05:30
Mayuri Nehate
1f3688a1ed
feat(ingest/databricks): include metadata for browse only tables ( #10766 )
...
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2024-09-02 19:09:45 +05:30
dushayntAW
92780e607c
fix(ingest/unity-catalog) upstream lineage for hive_metastore external table with s3 location ( #10546 )
2024-05-23 18:34:56 +05:30
Mayuri Nehate
529710ab9d
fix(ingest/databricks): handle and report config parse failure, updat… ( #10261 )
2024-04-17 12:14:16 -07:00
Mayuri Nehate
9928d70c1d
fix(ingest/databricks): support hive metastore schemas with special char ( #10049 )
2024-03-27 09:41:46 -07:00
dushayntAW
22487376de
fix(ingestion/unity-catalog): patch owners and properties ( #10086 )
2024-03-22 15:22:38 +05:30
dushayntAW
3a4bdef44a
fix(ingest/unity): generate sibling and lineage ( #9894 )
2024-03-18 20:22:17 +05:30
Mayuri Nehate
77df9ec926
feat(ingest/databricks): view upstream lineage for hive metastore ( #9657 )
2024-01-22 18:07:43 +05:30
Mayuri Nehate
f2e78db92e
feat(ingest/databricks): ingest hive metastore by default, more docs ( #9601 )
...
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2024-01-18 14:26:33 -08:00
Mayuri Nehate
29f2142a2c
feat(databricks): add hive metastore analyze profiling ( #9511 )
2024-01-02 23:18:43 -05:00
Mayuri Nehate
aac1c55a14
feat(ingest/unity): enable hive metastore ingestion ( #9416 )
2023-12-14 10:35:06 -05:00
Andrew Sikowitz
16fe22aafa
feat(ui/last-updated): Calculate last updated time as max(properties time, operation time) ( #9242 )
2023-12-06 15:21:56 -05:00
Andrew Sikowitz
5cb04981b2
test(ingest/unity): Update goldens ( #9254 )
2023-11-16 17:18:17 -05:00
Andrew Sikowitz
3a21c27f06
feat(ingest): Turn on browse path v2 creation ( #8342 )
...
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2023-07-06 16:43:42 -04:00
Andrew Sikowitz
584366771d
refactor(unity): Remove databricks_cli and cleanup ( #8249 )
...
Co-authored-by: Tamas Nemeth <treff7es@gmail.com>
2023-06-23 18:01:05 +05:30
Andrew Sikowitz
66806a805e
feat(ingest/unity): Set external url for containers and datasets ( #8238 )
2023-06-15 09:05:49 +02:00
Andrew Sikowitz
3022c2d12e
feat(ingest/unity): Add qualified name to dataset properties ( #8164 )
2023-06-05 11:20:13 -07:00
Andrew Sikowitz
d3cd4dbb0c
feat(ingest/unity): Allow ingestion without metastore admin role ( #8091 )
...
- Adds more detailed docs and connection test
- Fixes empty username queries
2023-05-24 15:36:22 -07:00
Andrew Sikowitz
5b290c9bc5
feat(ingest/unity): Add usage extraction; add TableReference ( #7910 )
...
- Adds usage extraction to the unity catalog source and a TableReference object to handle references to tables
Also makes the following refactors:
- Creates UsageAggregator class to usage_common, as I've seen this same logic multiple times.
- Allows customizable user_urn_builder in usage_common as not all unity users are emails. We create emails with a default email_domain config in other connectors like redshift and snowflake, which seems unnecessary now?
- Creates TableReference for unity catalog and adds it to the Table dataclass, for managing string references to tables. Replaces logic, especially in lineage extraction, with these references
- Creates gen_dataset_urn and gen_user_urn on unity source to reduce duplicate code
Breaks up proxy.py into implementation and types
2023-05-01 11:30:09 -07:00
Andrew Sikowitz
e9c2f9afcc
feat(ingest/unity): Ingest ownership for containers; lookup service principal display names ( #7869 )
2023-04-21 11:02:39 -07:00
Harshal Sheth
e461d03d94
feat(ingest/unity): capture create/lastModified timestamps ( #7819 )
...
Co-authored-by: Tamas Nemeth <treff7es@gmail.com>
2023-04-17 12:18:21 -07:00
Harshal Sheth
204727a6ee
feat(ingest/unity): support extracting ownership ( #7801 )
2023-04-12 19:45:41 -07:00
Tamas Nemeth
62e33e03a3
fix(ingest/unity): Use assigned metastore if not metastore listed in unity catalog ( #7446 )
2023-02-28 08:06:28 +01:00
Tamas Nemeth
0dc2d6af25
feat(ingest): databricks - Unity catalog source ( #6292 )
...
Co-authored-by: MohdSiddique Bagwan <mohdsiddique.bagwan@gslab.com>
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
2022-10-29 21:44:41 -07:00
