yangdx
1ca6837219
Add max nodes limit for graph retrieval of networkX
...
• Set MAX_GRAPH_NODES env var (default 1000)
• Change edge type to "RELATED"
2025-03-02 12:52:25 +08:00
yangdx
e3a40c2fdb
Fix linting
2025-03-01 16:23:34 +08:00
yangdx
d4f6dcfd54
Improve multi-process data synchronization and persistence in storage implementations
...
• Remove _get_client() or _get_graph() from index_done_callback
• Add return value for index_done_callback
2025-03-01 12:41:30 +08:00
yangdx
d3de57c1e4
Add multi-process support for vector database and graph storage with lock flags
...
• Implement storage lock mechanism
• Add update flag handling
• Add cross-process reload detection
2025-03-01 10:37:05 +08:00
yangdx
cd7648791a
Fix linting
2025-02-28 01:25:59 +08:00
yangdx
291e0c1b14
revert vector and graph use local data(single process)
2025-02-28 01:14:25 +08:00
yangdx
05d03638ec
Clean up logging output and remove redundant log messages
2025-02-27 20:17:28 +08:00
yangdx
64f22966a3
Fix linting
2025-02-27 19:05:51 +08:00
yangdx
1699b10a25
Refactor direct client/graph access to reduce redundant get calls in vector/graph ops
2025-02-27 15:14:54 +08:00
yangdx
f007ebf006
Refactor initialization logic for vector, KV and graph storage implementations
...
• Add try_initialize_namespace check
• Move init code out of storage locks
• Reduce redundant init conditions
• Simplify initialization flow
• Make init thread-safer
2025-02-27 14:55:07 +08:00
yangdx
7436c06f6c
Fix linting
2025-02-26 18:11:16 +08:00
yangdx
145bacc773
Add empty graph creation logging in NetworkXStorage
2025-02-26 17:42:30 +08:00
yangdx
2752a764ae
Refactor storage implementations to support both single and multi-process modes
...
• Add shared storage management module
• Support process/thread lock based on mode
2025-02-26 05:38:38 +08:00
yangdx
a642bb3190
refactor: use shared manager from main process for storage implementations.
2025-02-25 12:08:49 +08:00
yangdx
362321204f
Merge branch 'main' into add-multi-worker-support
2025-02-25 11:15:12 +08:00
yangdx
087d5770b0
feat(storage): Add shared memory support for file-based storage implementations
...
This commit adds multiprocessing shared memory support to file-based storage implementations:
- JsonDocStatusStorage
- JsonKVStorage
- NanoVectorDBStorage
- NetworkXStorage
Each storage module now uses module-level global variables with multiprocessing.Manager() to ensure data consistency across multiple uvicorn workers. All processes will see
updates immediately when data is modified through ainsert function.
2025-02-25 11:10:13 +08:00
yangdx
9fd0ab185f
Removed unnecessary comment
2025-02-24 16:28:18 +08:00
yangdx
b8543b8701
Remove unnecessary comment in networkx_impl.py
2025-02-24 09:55:03 +08:00
yangdx
9dd18ccd89
Simplify log message
2025-02-24 03:52:43 +08:00
yangdx
6cf555ebd5
Fix linting
2025-02-24 03:33:22 +08:00
yangdx
b4543561f6
Limit subgraph size to 500 nodes in NetworkXStorage
...
- Add max_graph_nodes check
- Reduce subgraph by degree
- Log graph size reduction
2025-02-24 03:32:33 +08:00
yangdx
b13c947360
Implement get_all_labels and get_knowledge_graph methods in NetworkXStorage
2025-02-24 02:37:42 +08:00
Yannick Stephan
678e0f9aea
Revert "Cleanup of code"
2025-02-20 15:09:43 +01:00
Yannick Stephan
439685e69c
Revert "removed get_knowledge_graph"
2025-02-20 14:29:36 +01:00
Yannick Stephan
c4562f71b9
cleanup extraction
2025-02-20 14:17:26 +01:00
Yannick Stephan
214e3e8ad5
fixed last update
2025-02-20 14:12:19 +01:00
Yannick Stephan
cedb5f3975
Merge pull request #886 from YanSte/clean-2
...
removed get_knowledge_graph
2025-02-19 22:59:10 +01:00
Yannick Stephan
89c35c82c7
removed get_knowledge_graph
2025-02-19 22:57:46 +01:00
Yannick Stephan
4978379b9c
Merge pull request #885 from YanSte/remove-unsused
...
Cleanup of code
2025-02-19 22:54:58 +01:00
Yannick Stephan
eb3306f34c
cleanup
2025-02-19 22:52:49 +01:00
Saifeddine ALOUI
d3c9af909b
Update networkx_impl.py
2025-02-19 18:40:13 +01:00
Yannick Stephan
495b0ddbe0
fixed networkx
2025-02-19 13:48:18 +01:00
Yannick Stephan
011227e426
cleanup no required install
2025-02-18 20:27:59 +01:00
Yannick Stephan
a0844bca28
cleaned import
2025-02-16 14:45:45 +01:00
Yannick Stephan
3fef8201c6
added final, required methods and cleaned import
2025-02-16 14:38:09 +01:00
Yannick Stephan
931c31fa8c
cleaned code
2025-02-16 13:55:30 +01:00
Yannick Stephan
882190a515
updated clean of what implemented on DocStatusStorage
2025-02-16 13:53:59 +01:00
zrguo
80451af839
fix linting errors
2025-01-27 23:21:34 +08:00
Saifeddine ALOUI
56e9c9f4d5
Moved the storages to kg folder
2025-01-27 09:59:26 +01:00
