Sergio Gómez Villamor
1c5b7c18fc
chore(ingestion): removes ignore for SIM117 ruff rule ( #13295 )
2025-04-23 15:55:46 +02:00
Andrew Sikowitz
92f013e6e1
fix(ingest/file-backed-collections): Properly set _use_sqlite_on_conflict ( #12297 )
2025-01-08 11:40:02 -08:00
Andrew Sikowitz
6b8d21a2ab
feat(ingest/sqlite): Support sqlite < 3.24.0 ( #12137 )
2024-12-16 12:50:25 -08:00
Harshal Sheth
311ea10833
feat(ingest): maintain ordering in file-backed dict ( #11346 )
2024-09-10 13:53:38 -07:00
Harshal Sheth
0d780e5f8f
feat(ingest): sql parsing aggregator ( #9786 )
2024-02-09 16:27:45 -05:00
Tamas Nemeth
1a47a51f1b
fix(ingest/build): Fix sagemaker mypy and flake8 issues ( #8530 )
2023-07-31 16:13:07 +02:00
Harshal Sheth
e99875cac6
chore(ingest): enable flake8 bugbear linting ( #7763 )
2023-04-10 14:14:42 -07:00
Andrew Sikowitz
ce1ac7fa12
refactor(ingest): Use sqlite.Row row_factory for FileBackedCollections ( #7739 )
2023-04-04 11:53:56 -07:00
Andrew Sikowitz
c7d35ffd66
perf(ingest): Improve FileBackedDict iteration performance; minor refactoring ( #7689 )
...
- Adds dirty bit to cache, only writes data if dirty
- Refactors __iter__
- Adds sql_query_iterator
- Adds items_snapshot, more performant `items()` that allows for filtering
- Renames connection -> shared_connection
- Removes unnecessary flush during close if connection is not shared
- Adds Closeable mixin
2023-03-27 17:20:34 -04:00
Andrew Sikowitz
8dd7a85533
refactor(ingest): Use shared connection wrapper over connection cache ( #7570 )
2023-03-14 15:09:37 -07:00
Harshal Sheth
fbfe43b1cb
feat(ingest): fix edge cases + interface cleanup for file-system APIs ( #7533 )
2023-03-13 13:14:53 -07:00
Harshal Sheth
b82afa89f1
feat(ingest): enable joins across FileBackedDicts + add FileBackedList ( #7506 )
2023-03-09 15:22:03 -08:00
Andrew Sikowitz
8101f0d47a
feat(ingest): Introduce FileBackedDict for offloading data to disk ( #7461 )
...
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
Also includes minor refactoring to the bigquery connector
2023-03-01 19:09:51 -05:00
