yangdx
d2708b966d
Added update flag to avoid persistence if no data is changed for KV storage
2025-03-10 01:17:25 +08:00
yangdx
4977c718f1
Improve KV storage initialize logic
2025-03-10 00:12:35 +08:00
yangdx
c938989920
Fix llm cache save problem in json_kv storage
2025-03-09 23:33:03 +08:00
yangdx
e47883d872
Add atomic data initialization lock to prevent race conditions
2025-03-09 17:33:15 +08:00
yangdx
c854aabde0
Add process ID to log messages for better multi-process debugging clarity
...
- Add PID to KV and Neo4j storage logs
- Add PID to query context logs
- Improve KV data count logging for llm cache
2025-03-09 15:25:10 +08:00
yangdx
90527875fd
Fix async issues in namespace init
2025-03-09 15:22:06 +08:00
yangdx
fd76e00c6a
Refactor storage initialization to separate object creation from data loading
...
• Split __post_init__ and initialize()
• Move data loading to initialize()
• Add FastAPI lifespan integration
2025-03-01 03:48:19 +08:00
yangdx
b3328542c7
refactor: migrate synchronous locks to async locks for improved concurrency
...
• Add UnifiedLock wrapper class
• Convert with blocks to async with
2025-03-01 02:22:35 +08:00
yangdx
cd7648791a
Fix linting
2025-02-28 01:25:59 +08:00
yangdx
05cf029bcc
fix: convert multiprocessing managed dict to normal dict before JSON dump
2025-02-27 20:16:53 +08:00
yangdx
7d12715f09
Refactor shared storage to safely handle multi-process initialization and data sharing
...
• Add namespace initialization check
• Use atomic operations for shared data
2025-02-26 18:11:02 +08:00
yangdx
2c019dbc7b
Refactor storage initialization to avoid redundant intitial data loads across processes, show init logs to first load only
2025-02-26 12:28:49 +08:00
yangdx
2752a764ae
Refactor storage implementations to support both single and multi-process modes
...
• Add shared storage management module
• Support process/thread lock based on mode
2025-02-26 05:38:38 +08:00
yangdx
a642bb3190
refactor: use shared manager from main process for storage implementations.
2025-02-25 12:08:49 +08:00
yangdx
087d5770b0
feat(storage): Add shared memory support for file-based storage implementations
...
This commit adds multiprocessing shared memory support to file-based storage implementations:
- JsonDocStatusStorage
- JsonKVStorage
- NanoVectorDBStorage
- NetworkXStorage
Each storage module now uses module-level global variables with multiprocessing.Manager() to ensure data consistency across multiple uvicorn workers. All processes will see
updates immediately when data is modified through ainsert function.
2025-02-25 11:10:13 +08:00
yangdx
043dbad683
Merge remote-tracking branch 'origin/main' into refactor-api-server
2025-02-21 11:24:16 +08:00
Yannick Stephan
9277fe8c29
fixed return
2025-02-19 22:22:41 +01:00
yangdx
f776db0779
Improved document status retrieval with content fallback.
...
- Added content fallback to content_summary
- Handled missing fields gracefully
- Made data copy to avoid modification
- Added error logging for missing fields
- Improved code readability and robustness
2025-02-20 04:09:02 +08:00
Saifeddine ALOUI
1fe47e5ef1
Update json_doc_status_impl.py
2025-02-18 10:24:54 +01:00
Saifeddine ALOUI
20909e495b
Added drop
2025-02-18 10:22:16 +01:00
Yannick Stephan
80272cbf16
fixed edge
2025-02-17 23:20:10 +01:00
Yannick Stephan
66c4b01fdd
remove drops unused
2025-02-17 23:16:23 +01:00
Yannick Stephan
abad9f235c
make value on str enum
2025-02-16 15:54:54 +01:00
Yannick Stephan
2bf238396e
updated wrong status
2025-02-16 15:52:59 +01:00
Yannick Stephan
0a8c94a1e0
fix value
2025-02-16 14:51:24 +01:00
Yannick Stephan
9a5fbaaa5f
removed unused methods
2025-02-16 14:50:04 +01:00
Yannick Stephan
3fef8201c6
added final, required methods and cleaned import
2025-02-16 14:38:09 +01:00
ArnoChen
42d90f2b16
rename jsondocstatus to json_doc_status for consistency
2025-02-13 18:38:12 +08:00
