yangdx
e56734cb8b
Refac: Optimize document deletion performance
...
- Adding chunks_list to dock_status
- Adding llm_cache_list to text_chunks
- Implemented storage types: JsonKV and Redis
2025-07-03 04:18:25 +08:00
yangdx
bef7206192
Optimize logger info
2025-04-28 02:27:59 +08:00
yangdx
ad087073aa
Optimize logger for storage
2025-04-10 01:07:06 +08:00
yangdx
ff5c7182da
Fix update status handling bugs in drop function of json kv storage
2025-04-01 13:53:02 +08:00
yangdx
95a8ee27ed
Fix linting
2025-03-31 23:22:27 +08:00
yangdx
3d4f8f67c9
Add drop_cace_by_modes to all KV storage implementation
2025-03-31 23:10:21 +08:00
yangdx
1772e7a887
Add delete support to all storage implementation
2025-03-31 16:21:20 +08:00
yangdx
2cb64ad280
feat: Remove immediate persistence in delete operation for JsonDocStatusStorage
2025-03-31 14:46:36 +08:00
yangdx
1df4b777d7
Add drop funtions to storage implementations
2025-03-30 15:17:57 +08:00
yangdx
20de4ded30
Add default file_path for missing document paths
...
- Set file_path to "no-file-path" if missing
- Ensure consistent document data structure
2025-03-18 20:06:18 +08:00
yangdx
46610682ce
Fix data persistence issue in single-process mode
...
In single-process mode, data updates and persistence were not working properly because the update flags were not being correctly handled between different objects.
2025-03-10 15:41:00 +08:00
yangdx
4065a7df92
Fix linting
2025-03-10 02:07:19 +08:00
yangdx
14e1b31d1c
Improved logging clarity in storage operations
2025-03-10 02:05:55 +08:00
yangdx
6b0acce644
Avoid redundant llm cache updates
2025-03-10 01:45:58 +08:00
yangdx
d2708b966d
Added update flag to avoid persistence if no data is changed for KV storage
2025-03-10 01:17:25 +08:00
yangdx
4977c718f1
Improve KV storage initialize logic
2025-03-10 00:12:35 +08:00
yangdx
c938989920
Fix llm cache save problem in json_kv storage
2025-03-09 23:33:03 +08:00
yangdx
e47883d872
Add atomic data initialization lock to prevent race conditions
2025-03-09 17:33:15 +08:00
yangdx
c854aabde0
Add process ID to log messages for better multi-process debugging clarity
...
- Add PID to KV and Neo4j storage logs
- Add PID to query context logs
- Improve KV data count logging for llm cache
2025-03-09 15:25:10 +08:00
yangdx
90527875fd
Fix async issues in namespace init
2025-03-09 15:22:06 +08:00
yangdx
fd76e00c6a
Refactor storage initialization to separate object creation from data loading
...
• Split __post_init__ and initialize()
• Move data loading to initialize()
• Add FastAPI lifespan integration
2025-03-01 03:48:19 +08:00
yangdx
b3328542c7
refactor: migrate synchronous locks to async locks for improved concurrency
...
• Add UnifiedLock wrapper class
• Convert with blocks to async with
2025-03-01 02:22:35 +08:00
yangdx
cd7648791a
Fix linting
2025-02-28 01:25:59 +08:00
yangdx
05cf029bcc
fix: convert multiprocessing managed dict to normal dict before JSON dump
2025-02-27 20:16:53 +08:00
yangdx
7d12715f09
Refactor shared storage to safely handle multi-process initialization and data sharing
...
• Add namespace initialization check
• Use atomic operations for shared data
2025-02-26 18:11:02 +08:00
yangdx
2c019dbc7b
Refactor storage initialization to avoid redundant intitial data loads across processes, show init logs to first load only
2025-02-26 12:28:49 +08:00
yangdx
2752a764ae
Refactor storage implementations to support both single and multi-process modes
...
• Add shared storage management module
• Support process/thread lock based on mode
2025-02-26 05:38:38 +08:00
yangdx
a642bb3190
refactor: use shared manager from main process for storage implementations.
2025-02-25 12:08:49 +08:00
yangdx
087d5770b0
feat(storage): Add shared memory support for file-based storage implementations
...
This commit adds multiprocessing shared memory support to file-based storage implementations:
- JsonDocStatusStorage
- JsonKVStorage
- NanoVectorDBStorage
- NetworkXStorage
Each storage module now uses module-level global variables with multiprocessing.Manager() to ensure data consistency across multiple uvicorn workers. All processes will see
updates immediately when data is modified through ainsert function.
2025-02-25 11:10:13 +08:00
yangdx
043dbad683
Merge remote-tracking branch 'origin/main' into refactor-api-server
2025-02-21 11:24:16 +08:00
Yannick Stephan
9277fe8c29
fixed return
2025-02-19 22:22:41 +01:00
yangdx
f776db0779
Improved document status retrieval with content fallback.
...
- Added content fallback to content_summary
- Handled missing fields gracefully
- Made data copy to avoid modification
- Added error logging for missing fields
- Improved code readability and robustness
2025-02-20 04:09:02 +08:00
Saifeddine ALOUI
1fe47e5ef1
Update json_doc_status_impl.py
2025-02-18 10:24:54 +01:00
Saifeddine ALOUI
20909e495b
Added drop
2025-02-18 10:22:16 +01:00
Yannick Stephan
80272cbf16
fixed edge
2025-02-17 23:20:10 +01:00
Yannick Stephan
66c4b01fdd
remove drops unused
2025-02-17 23:16:23 +01:00
Yannick Stephan
abad9f235c
make value on str enum
2025-02-16 15:54:54 +01:00
Yannick Stephan
2bf238396e
updated wrong status
2025-02-16 15:52:59 +01:00
Yannick Stephan
0a8c94a1e0
fix value
2025-02-16 14:51:24 +01:00
Yannick Stephan
9a5fbaaa5f
removed unused methods
2025-02-16 14:50:04 +01:00
Yannick Stephan
3fef8201c6
added final, required methods and cleaned import
2025-02-16 14:38:09 +01:00
ArnoChen
42d90f2b16
rename jsondocstatus to json_doc_status for consistency
2025-02-13 18:38:12 +08:00
