17 Commits

Author SHA1 Message Date
skrydal
47490ec050
feat(ingestion/iceberg): Add capability to extract namespace properties to the iceberg ingestor (#13238) 2025-04-17 16:29:43 +02:00
skrydal
38f1553315
feat(ingestion): Refactoring timestamping logic for WorkUnits + custom logic for Iceberg (#13030)
Co-authored-by: Sergio Gómez Villamor <sgomezvillamor@gmail.com>
2025-04-04 22:30:27 +02:00
Sergio Gómez Villamor
b37fa03846
chore: fixes SIM118 ruff rule (#13069) 2025-04-04 11:59:43 +02:00
skrydal
b7bb0b056c
feat(ingestion/iceberg): Refactor iceberg connector (#12921) 2025-03-28 18:34:36 +01:00
skrydal
1185ba8121
feat(ingestion/iceberg): Refactor iceberg source to use MCPWs instead of MCEs (#12912) 2025-03-20 10:46:08 +01:00
skrydal
3e1b20c727
feat(ingestion/iceberg): Several improvements to iceberg connector (#12744) 2025-03-01 01:30:07 +01:00
Aseem Bansal
d1494c2252
dev: upgrade ruff, remove 2 ruff ignore rules (#12702) 2025-02-24 14:07:57 +05:30
Aseem Bansal
262dd76518
dev: remove black in favor of ruff for formatting (#12378) 2025-01-18 15:06:20 +05:30
skrydal
e45f548910
feat(ingest/iceberg): Improve iceberg connector (#12163) 2024-12-19 15:25:59 +00:00
skrydal
2527f54972
feat(ingest/iceberg): Iceberg performance improvement (multi-threading) (#11182) 2024-11-18 19:41:45 +01:00
Eric L (CCCS)
79ba0b1720
fix(ingest/iceberg): add support for nested dictionaries when configuring pyiceberg (#10762) 2024-06-21 14:38:01 -07:00
Eric L (CCCS)
c04b3bc2e4
fix(ingest/iceberg): update iceberg source to support newer versions of pyiceberg at runtime (#10614) 2024-06-04 09:45:29 -07:00
Harshal Sheth
1498c36875
chore(cli): drop support for python 3.7 (#9731) 2024-01-29 10:50:47 -08:00
cccs-eric
6fe60a274e
feat(iceberg): Upgrade Iceberg ingestion source to pyiceberg 0.4.0 (#8357)
Co-authored-by: cccs-Dustin <96579982+cccs-Dustin@users.noreply.github.com>
Co-authored-by: Fokko Driesprong <fokko@apache.org>
Co-authored-by: Andrew Sikowitz <andrew.sikowitz@acryl.io>
2023-08-31 13:01:05 -04:00
Harshal Sheth
355c129c7c
chore(ingest): drop python 3.6 support (#5521) 2022-08-10 15:00:31 -07:00
cccs-eric
f25a5f043d
fix(iceberg): Change how MapType are mapped to Avro to support complex Map key type. (#5060) 2022-06-09 08:13:10 +05:30
Ravindra Lanka
108b492ed1
feat(ingestion): Add Iceberg source (#5010)
Co-authored-by: cccs-eric <eric.ladouceur@cyber.gc.ca>
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
2022-05-26 08:05:57 -07:00