zrguo
91d0f65476
Update QueryParam
2025-07-15 14:21:58 +08:00
yangdx
3da9f8aab4
Fix logging output condition in shared_storage.py. Early return if logging disabled
2025-07-15 13:38:05 +08:00
yangdx
85cd1178a1
fix: prevent premature lock cleanup in multiprocess mode
...
- Change cleanup condition from count == 1 to count == 0 to properly
remove reused locks from cleanup list
- Fix RuntimeError: Attempting to release lock for xxxx more times than it was acquired
2025-07-13 13:51:48 +08:00
yangdx
a2eeae9661
Fixes incorrect cleanup count
2025-07-13 02:38:36 +08:00
yangdx
582e952020
Disable direct logging by default for shared storage module
2025-07-13 01:58:50 +08:00
yangdx
cbf544b3c1
Remvoe redundant log message
2025-07-13 01:51:30 +08:00
yangdx
0e3aaa318f
Feat: Add keyed lock cleanup and status monitoring
2025-07-13 00:09:00 +08:00
yangdx
2ade3067f8
Refac: Generalize keyed lock with namespace support
...
Refactored the `KeyedUnifiedLock` to be generic and support dynamic namespaces. This decouples the locking mechanism from a specific "GraphDB" implementation, allowing it to be reused across different components and workspaces safely.
Key changes:
- `KeyedUnifiedLock` now takes a `namespace` parameter on lock acquisition.
- Renamed `_graph_db_lock_keyed` to a more generic _storage_keyed_lock`
- Replaced `get_graph_db_lock_keyed` with get_storage_keyed_lock` to support namespaces
2025-07-12 12:10:12 +08:00
yangdx
f2d875f8ab
Update comments
2025-07-12 11:05:25 +08:00
yangdx
5ee509e671
Fix linting
2025-07-12 05:17:44 +08:00
yangdx
964293f21b
Optimize lock cleanup with time tracking and intervals
...
- Add cleanup time tracking variables
- Implement minimum cleanup intervals
- Track earliest cleanup times
- Handle time rollback cases
- Improve cleanup logging
2025-07-12 04:34:26 +08:00
yangdx
39965d7ded
Move merging stage back controled by max parallel insert semhore
2025-07-12 03:32:08 +08:00
yangdx
7490a18481
Optimize lock cleanup parameters
2025-07-12 03:10:03 +08:00
yangdx
3d8e6924bc
Show lock clean up message
2025-07-12 02:58:05 +08:00
yangdx
22c36f2fd2
Optimize log messages
2025-07-12 02:41:31 +08:00
yangdx
a64c767298
optimize: improve lock cleanup performance with threshold-based strategy
...
- Add CLEANUP_THRESHOLD constant (100) to control cleanup frequency
- Modify _release_shared_raw_mp_lock to only scan when cleanup list exceeds threshold
- Modify _release_async_lock to only scan when cleanup list exceeds threshold
2025-07-11 23:43:40 +08:00
yangdx
ad99d9ba5a
Improve code organization and comments
2025-07-11 22:13:02 +08:00
yangdx
c52c451cf7
Fix linting
2025-07-11 20:40:50 +08:00
yangdx
3afdd1b67c
Fix initial count error for multi-process lock with key
2025-07-11 20:39:08 +08:00
Arjun Rao
f8149790e4
Initial commit with keyed graph lock
2025-05-08 12:29:49 +10:00
yangdx
9f33ff2ecd
Optimize log messages
2025-04-29 13:45:06 +08:00
yangdx
5f3e210246
Optimize log messages
2025-04-29 13:32:05 +08:00
yangdx
3aef63cc65
Optimize log info
2025-04-28 23:17:09 +08:00
yangdx
922fc914be
Change empty pipeline job name
2025-03-26 17:48:00 +08:00
yangdx
15e060f854
Fix share storage update status handling problem of in memeory storage
2025-03-25 10:48:15 +08:00
yangdx
53396e4d82
Fixlinting
2025-03-21 16:56:47 +08:00
yangdx
20d65ae554
feat(shared_storage): prevent event loop blocking in multiprocess mode
...
Add auxiliary async locks in multiprocess mode to prevent event loop blocking
2025-03-21 16:08:23 +08:00
yangdx
5d64f3b0a0
Improved auto-scan task initialization and status tracking.
...
- Added autoscan status tracking in pipeline
- Ensured auto-scan runs only once per startup
2025-03-10 17:14:14 +08:00
yangdx
57a41eedb8
Fix linting
2025-03-10 15:41:46 +08:00
yangdx
46610682ce
Fix data persistence issue in single-process mode
...
In single-process mode, data updates and persistence were not working properly because the update flags were not being correctly handled between different objects.
2025-03-10 15:41:00 +08:00
yangdx
4065a7df92
Fix linting
2025-03-10 02:07:19 +08:00
yangdx
d2708b966d
Added update flag to avoid persistence if no data is changed for KV storage
2025-03-10 01:17:25 +08:00
yangdx
e47883d872
Add atomic data initialization lock to prevent race conditions
2025-03-09 17:33:15 +08:00
yangdx
90527875fd
Fix async issues in namespace init
2025-03-09 15:22:06 +08:00
yangdx
c5d0962872
Fix linting
2025-03-09 01:00:42 +08:00
yangdx
95c06f1bde
Add graph DB lock to shared storage system
...
• Introduced new graph_db_lock
• Added detailed lock debugging output
2025-03-08 22:36:41 +08:00
yangdx
7cd25fe5ab
Improve shared storage cleanup and clarify initialization in multi-worker setup
2025-03-02 01:00:27 +08:00
yangdx
e3a40c2fdb
Fix linting
2025-03-01 16:23:34 +08:00
yangdx
40e9e26edb
feat: add update flags status to API health endpoint
2025-03-01 14:58:26 +08:00
yangdx
c07a5039b7
Refactor shared storage locks to separate pipeline, storage and internal locks for deadlock preventing
2025-03-01 10:48:55 +08:00
yangdx
d704512139
Refactor shared storage module to improve async handling and naming consistency
...
• Add async support for get_namespace_data
• Rename get_update_flags to get_update_flag
• Rename set_update_flag to set_all_update_flags
• Update docstrings for clarity
• Fix typos in log messages
2025-03-01 05:01:26 +08:00
yangdx
fd76e00c6a
Refactor storage initialization to separate object creation from data loading
...
• Split __post_init__ and initialize()
• Move data loading to initialize()
• Add FastAPI lifespan integration
2025-03-01 03:48:19 +08:00
yangdx
b3328542c7
refactor: migrate synchronous locks to async locks for improved concurrency
...
• Add UnifiedLock wrapper class
• Convert with blocks to async with
2025-03-01 02:22:35 +08:00
yangdx
a721421bd8
Add async support and update flag mechanism for shared storage
...
• Use asyncio.Lock instead of thread lock for single process mode
• Add storage update notification system
2025-03-01 01:49:26 +08:00
yangdx
731d820bcc
Remove redundancy set_logger function and related calls
2025-02-28 21:46:45 +08:00
yangdx
8cd45161f2
feat: add history_messages to track pipeline processing progress
...
• Add shared history_messages list
• Track pipeline progress with messages
2025-02-28 13:53:40 +08:00
yangdx
b090a22be7
Add concurrency check for auto scan task to prevent duplicate scans
...
• Add pipeline status check before scan
• Add storage lock protection
• Add latest_message to status tracking
• Add helpful log message at startup
2025-02-28 12:22:20 +08:00
yangdx
b2da69b7f1
Add pipeline status control for concurrent document indexing processes
...
• Add shared pipeline status namespace
• Implement concurrent process control
• Add request queuing for pending jobs
2025-02-28 11:52:42 +08:00
yangdx
cd7648791a
Fix linting
2025-02-28 01:25:59 +08:00
yangdx
3dcfa561d7
Remove debug logging
2025-02-28 01:15:12 +08:00
