yangdx
5d64f3b0a0
Improved auto-scan task initialization and status tracking.
...
- Added autoscan status tracking in pipeline
- Ensured auto-scan runs only once per startup
2025-03-10 17:14:14 +08:00
yangdx
57a41eedb8
Fix linting
2025-03-10 15:41:46 +08:00
yangdx
46610682ce
Fix data persistence issue in single-process mode
...
In single-process mode, data updates and persistence were not working properly because the update flags were not being correctly handled between different objects.
2025-03-10 15:41:00 +08:00
yangdx
4065a7df92
Fix linting
2025-03-10 02:07:19 +08:00
yangdx
d2708b966d
Added update flag to avoid persistence if no data is changed for KV storage
2025-03-10 01:17:25 +08:00
yangdx
e47883d872
Add atomic data initialization lock to prevent race conditions
2025-03-09 17:33:15 +08:00
yangdx
90527875fd
Fix async issues in namespace init
2025-03-09 15:22:06 +08:00
yangdx
c5d0962872
Fix linting
2025-03-09 01:00:42 +08:00
yangdx
95c06f1bde
Add graph DB lock to shared storage system
...
• Introduced new graph_db_lock
• Added detailed lock debugging output
2025-03-08 22:36:41 +08:00
yangdx
7cd25fe5ab
Improve shared storage cleanup and clarify initialization in multi-worker setup
2025-03-02 01:00:27 +08:00
yangdx
e3a40c2fdb
Fix linting
2025-03-01 16:23:34 +08:00
yangdx
40e9e26edb
feat: add update flags status to API health endpoint
2025-03-01 14:58:26 +08:00
yangdx
c07a5039b7
Refactor shared storage locks to separate pipeline, storage and internal locks for deadlock preventing
2025-03-01 10:48:55 +08:00
yangdx
d704512139
Refactor shared storage module to improve async handling and naming consistency
...
• Add async support for get_namespace_data
• Rename get_update_flags to get_update_flag
• Rename set_update_flag to set_all_update_flags
• Update docstrings for clarity
• Fix typos in log messages
2025-03-01 05:01:26 +08:00
yangdx
fd76e00c6a
Refactor storage initialization to separate object creation from data loading
...
• Split __post_init__ and initialize()
• Move data loading to initialize()
• Add FastAPI lifespan integration
2025-03-01 03:48:19 +08:00
yangdx
b3328542c7
refactor: migrate synchronous locks to async locks for improved concurrency
...
• Add UnifiedLock wrapper class
• Convert with blocks to async with
2025-03-01 02:22:35 +08:00
yangdx
a721421bd8
Add async support and update flag mechanism for shared storage
...
• Use asyncio.Lock instead of thread lock for single process mode
• Add storage update notification system
2025-03-01 01:49:26 +08:00
yangdx
731d820bcc
Remove redundancy set_logger function and related calls
2025-02-28 21:46:45 +08:00
yangdx
8cd45161f2
feat: add history_messages to track pipeline processing progress
...
• Add shared history_messages list
• Track pipeline progress with messages
2025-02-28 13:53:40 +08:00
yangdx
b090a22be7
Add concurrency check for auto scan task to prevent duplicate scans
...
• Add pipeline status check before scan
• Add storage lock protection
• Add latest_message to status tracking
• Add helpful log message at startup
2025-02-28 12:22:20 +08:00
yangdx
b2da69b7f1
Add pipeline status control for concurrent document indexing processes
...
• Add shared pipeline status namespace
• Implement concurrent process control
• Add request queuing for pending jobs
2025-02-28 11:52:42 +08:00
yangdx
cd7648791a
Fix linting
2025-02-28 01:25:59 +08:00
yangdx
3dcfa561d7
Remove debug logging
2025-02-28 01:15:12 +08:00
yangdx
291e0c1b14
revert vector and graph use local data(single process)
2025-02-28 01:14:25 +08:00
yangdx
05d03638ec
Clean up logging output and remove redundant log messages
2025-02-27 20:17:28 +08:00
yangdx
64f22966a3
Fix linting
2025-02-27 19:05:51 +08:00
yangdx
946095ef80
Fix multiprocess dict creation logic, add process safety locks for namespace creation.
2025-02-27 19:03:53 +08:00
yangdx
e881bc0709
simplify process state management by removing redundant multiprocess flag
2025-02-27 15:36:12 +08:00
yangdx
03d05b094d
Improve Gunicorn support and cleanup shared storage initialization
...
• Move Gunicorn check before other startup
• Improve startup flow organization
2025-02-27 14:13:42 +08:00
yangdx
7aec78833c
Implement Gunicorn+Uvicorn integration for shared data preloading
...
- Create run_with_gunicorn.py script to properly initialize shared data in the
main process before forking worker processes
- Revert unvicorn to single process mode only, and let gunicorn do all the multi-process jobs
2025-02-27 13:25:22 +08:00
yangdx
7c237920b1
Refactor shared storage to support both single and multi-process modes
...
• Initialize storage based on worker count
• Remove redundant global variable checks
• Add explicit mutex initialization
• Centralize shared storage initialization
• Fix process/thread lock selection logic
2025-02-27 08:48:33 +08:00
yangdx
7436c06f6c
Fix linting
2025-02-26 18:11:16 +08:00
yangdx
7d12715f09
Refactor shared storage to safely handle multi-process initialization and data sharing
...
• Add namespace initialization check
• Use atomic operations for shared data
2025-02-26 18:11:02 +08:00
yangdx
2752a764ae
Refactor storage implementations to support both single and multi-process modes
...
• Add shared storage management module
• Support process/thread lock based on mode
2025-02-26 05:38:38 +08:00
