Harshal Sheth
38bcd9c381
feat(ingest): default to ASYNC_BATCH mode in datahub-rest sink ( #11369 )
2024-09-17 07:11:58 +01:00
Harshal Sheth
311ea10833
feat(ingest): maintain ordering in file-backed dict ( #11346 )
2024-09-10 13:53:38 -07:00
Felix Lüdin
9619553e2d
fix(ingest): use correct native data type in all SQLAlchemy sources by compiling data type using dialect ( #10898 )
...
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2024-08-06 12:52:20 -07:00
sid-acryl
dffdef2eaa
fix(ingestion/powerbi): fix issue with broken report lineage ( #10910 )
2024-07-31 11:40:09 -07:00
Mayuri Nehate
ff1c6b895e
feat(ingest/BigQuery): refactor+parallelize dataset metadata extraction ( #10884 )
2024-07-16 11:46:42 -07:00
Harshal Sheth
0d677e4992
fix(ingest/snowflake): fix column batcher ( #10781 )
2024-06-25 22:21:54 -07:00
Harshal Sheth
724907b8f4
feat(ingest): add async batch mode to the rest sink ( #10733 )
2024-06-25 15:49:00 -07:00
Harshal Sheth
0dc0bc5761
feat(ingest/snowflake): performance improvements ( #10746 )
2024-06-25 14:46:55 -07:00
Harshal Sheth
3d5735cbc5
chore(ingest): run pyupgrade for python 3.8 ( #10513 )
2024-05-15 22:31:05 -07:00
siladitya
43ac405c57
fix(metadata-ingestion)glue connector failure when Optional field Type of PartitionKey is absent for a Table ( #10052 )
2024-03-20 11:02:28 +01:00
Harshal Sheth
92a3ac6f11
fix(ingest): use contextvar for cooperative timeout ( #10021 )
2024-03-11 14:14:39 -07:00
Harshal Sheth
0d780e5f8f
feat(ingest): sql parsing aggregator ( #9786 )
2024-02-09 16:27:45 -05:00
Harshal Sheth
98e3da42f5
feat(ingest/looker): add backpressure-aware executor ( #9615 )
...
Co-authored-by: Tamas Nemeth <treff7es@gmail.com>
2024-01-12 18:54:08 +01:00
Harshal Sheth
f05056aed7
feat(ingest): key-partitioning for rest emitter ( #9613 )
2024-01-11 17:57:48 -05:00
Harshal Sheth
7517c77ffd
fix(ingest): resolve issue with caplog and asyncio ( #9377 )
2023-12-04 20:00:11 -05:00
Andrew Sikowitz
adf8c8db38
refactor(ingest): Move sqlalchemy import out of sql_types.py ( #9065 )
2023-10-24 08:59:56 +02:00
Tim
1eaf9c8c5f
feature(ingest/athena): introduce support for complex and nested schemas in Athena ( #8137 )
...
Co-authored-by: dnks23 <dominik.s23@live.de>
Co-authored-by: Tamas Nemeth <treff7es@gmail.com>
Co-authored-by: Tim <tim@MBP-von-Tim.fritz.box>
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2023-10-18 09:39:59 -07:00
Mayuri Nehate
c81a339bfc
build(ingest): remove ratelimiter dependency ( #9008 )
2023-10-16 09:27:57 -07:00
Mayuri Nehate
cdb9f5ba62
feat(bigquery): add better timers around every API call ( #8626 )
2023-09-15 11:55:39 -07:00
Tamas Nemeth
1a47a51f1b
fix(ingest/build): Fix sagemaker mypy and flake8 issues ( #8530 )
2023-07-31 16:13:07 +02:00
Harshal Sheth
08d4e904a8
feat(ingest): add YamlFileUpdater utility ( #8266 )
2023-06-29 13:15:34 -07:00
Harshal Sheth
e99875cac6
chore(ingest): enable flake8 bugbear linting ( #7763 )
2023-04-10 14:14:42 -07:00
Andrew Sikowitz
ce1ac7fa12
refactor(ingest): Use sqlite.Row row_factory for FileBackedCollections ( #7739 )
2023-04-04 11:53:56 -07:00
Andrew Sikowitz
c7d35ffd66
perf(ingest): Improve FileBackedDict iteration performance; minor refactoring ( #7689 )
...
- Adds dirty bit to cache, only writes data if dirty
- Refactors __iter__
- Adds sql_query_iterator
- Adds items_snapshot, more performant `items()` that allows for filtering
- Renames connection -> shared_connection
- Removes unnecessary flush during close if connection is not shared
- Adds Closeable mixin
2023-03-27 17:20:34 -04:00
Andrew Sikowitz
8dd7a85533
refactor(ingest): Use shared connection wrapper over connection cache ( #7570 )
2023-03-14 15:09:37 -07:00
Harshal Sheth
fbfe43b1cb
feat(ingest): fix edge cases + interface cleanup for file-system APIs ( #7533 )
2023-03-13 13:14:53 -07:00
Harshal Sheth
b82afa89f1
feat(ingest): enable joins across FileBackedDicts + add FileBackedList ( #7506 )
2023-03-09 15:22:03 -08:00
Andrew Sikowitz
8101f0d47a
feat(ingest): Introduce FileBackedDict for offloading data to disk ( #7461 )
...
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
Also includes minor refactoring to the bigquery connector
2023-03-01 19:09:51 -05:00
Tamas Nemeth
9015a43f25
fix(ingest): bigquery-beta - Adding python 3.8 fix for memory footprint util ( #6228 )
2022-10-18 17:59:31 -07:00
Tamas Nemeth
2f79b50c24
fix(ingest): presto-on-hive - not failing on Hive type parsing error ( #6118 )
...
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
2022-10-04 20:54:38 -07:00
Mayuri Nehate
b195b6c123
fix(ingest): encode reserved characters when creating dataset urn ( #5977 )
...
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2022-09-20 16:59:02 -07:00
Shirshanka Das
9afda47085
feat(cli): add support for sampled reporting to keep logs manageable ( #5800 )
2022-09-01 14:47:28 -07:00
