yangdx
946095ef80
Fix multiprocess dict creation logic, add process safety locks for namespace creation.
2025-02-27 19:03:53 +08:00
yangdx
92ecb0da97
Refactor document scanning progress share variable initialization
2025-02-27 16:07:00 +08:00
yangdx
7c237920b1
Refactor shared storage to support both single and multi-process modes
...
• Initialize storage based on worker count
• Remove redundant global variable checks
• Add explicit mutex initialization
• Centralize shared storage initialization
• Fix process/thread lock selection logic
2025-02-27 08:48:33 +08:00
yangdx
7436c06f6c
Fix linting
2025-02-26 18:11:16 +08:00
yangdx
2752a764ae
Refactor storage implementations to support both single and multi-process modes
...
• Add shared storage management module
• Support process/thread lock based on mode
2025-02-26 05:38:38 +08:00
yangdx
8050b0f91b
feat: automatically initialize API manager in single process mode
...
- Add manager init check in __post_init__
- Call initialize_manager if needed
- Add info log message for init
- Ensure API manager is ready for use
2025-02-25 12:09:30 +08:00
yangdx
f29628125b
Fix typo in parameter name from 'nodel_label' to 'node_label'
2025-02-24 02:36:36 +08:00
yangdx
f5efe5977b
Merge branch 'clear-text-before-insert' into simplify-cli-arguments
2025-02-23 17:06:39 +08:00
yangdx
845e914f1b
fix: make ids parameter optional and optimize input text cleaning
...
- Add default None value for ids parameter
- Move text cleaning into else branch
- Only clean text when auto-generating ids
- Preserve original text with custom ids
- Improve code readability
2025-02-23 15:46:47 +08:00
yangdx
e935fed50e
Add automatic comment handling in .env files
2025-02-22 13:25:12 +08:00
yangdx
351c8db849
Fix linting
2025-02-22 10:27:20 +08:00
yangdx
411782797b
Fix linting
2025-02-22 10:18:39 +08:00
yangdx
3c866eec16
Merge branch 'refactor-api-server' into clear-text-before-insert
2025-02-22 10:04:56 +08:00
yangdx
dff07e50a4
Merge branch 'main' into refactor-api-server
2025-02-21 21:12:02 +08:00
zrguo
6ed81ed1c6
Merge pull request #906 from konrad-woj/fix-insert-custom-chunks
...
fix insert_custom_chunks skipping every new doc
2025-02-21 18:45:40 +08:00
zrguo
84f975f63f
Merge pull request #892 from PiochU19/main
...
add support of providing ids for documents insert
2025-02-21 18:42:52 +08:00
yangdx
5fa6982d36
Merge branch 'refactor-api-server' into clear-text-before-insert
2025-02-21 14:57:11 +08:00
yangdx
cff229a806
fix: respect user-specified log level in set_logger
...
Previously, the set_logger function would always set the log level to DEBUG, overriding any user-specified log level.
2025-02-21 14:46:27 +08:00
yangdx
f5bd3f2b16
Fix linting
2025-02-21 13:23:55 +08:00
yangdx
bee4622052
fix: handle null bytes (0x00) in text processing
...
- Fix PostgreSQL encoding error by properly handling null bytes (0x00) in text processing.
- The clean_text function now removes null bytes from all input text during the indexing phase.
2025-02-21 13:18:26 +08:00
Konrad Wojciechowski
50eb97762a
fix insert_custom_chunks skipping every new doc with "This document is already in the storage."
2025-02-20 23:08:36 +01:00
Yannick Stephan
678e0f9aea
Revert "Cleanup of code"
2025-02-20 15:09:43 +01:00
Yannick Stephan
439685e69c
Revert "removed get_knowledge_graph"
2025-02-20 14:29:36 +01:00
Yannick Stephan
c4562f71b9
cleanup extraction
2025-02-20 14:17:26 +01:00
Yannick Stephan
214e3e8ad5
fixed last update
2025-02-20 14:12:19 +01:00
Yannick Stephan
3e820cc68e
fixed default factory
2025-02-20 14:04:59 +01:00
PiochU19
d462ace978
add support of providing ids for documents insert
2025-02-20 14:02:52 +01:00
Yannick Stephan
8413537ad2
cleanup
2025-02-20 13:44:17 +01:00
Yannick Stephan
4923f5c1d7
cleanup kg
2025-02-20 13:39:46 +01:00
Yannick Stephan
60717fd6be
cleanup storage state
2025-02-20 13:30:30 +01:00
Yannick Stephan
59bb75d4a1
added log path
2025-02-20 13:27:55 +01:00
Yannick Stephan
c7bc2c63cf
cleanup storages
2025-02-20 13:21:41 +01:00
Yannick Stephan
32e489865c
cleanup code
2025-02-20 13:18:17 +01:00
Yannick Stephan
72b978d6d5
cleanup
2025-02-20 13:13:38 +01:00
Yannick Stephan
32d0f1acb0
added docs and fields
2025-02-20 13:09:33 +01:00
Yannick Stephan
4b478d1c0f
added fields
2025-02-20 13:06:34 +01:00
Yannick Stephan
f5a93c7bb5
added fields
2025-02-20 13:06:16 +01:00
Yannick Stephan
2370a4336b
added field
2025-02-20 13:05:59 +01:00
Yannick Stephan
37addb7c01
added final
2025-02-20 13:05:35 +01:00
Yannick Stephan
bae21a6fad
added max paralle insert
2025-02-20 12:57:25 +01:00
Yannick Stephan
de56aeb7c5
removed lock
2025-02-20 12:54:52 +01:00
Yannick Stephan
26abae4733
Merge pull request #891 from YanSte/lock
...
added lock
2025-02-20 00:21:28 +01:00
Yannick Stephan
fa99d72269
added lock
2025-02-20 00:20:30 +01:00
Yannick Stephan
bf720b131b
Merge pull request #890 from YanSte/paralle-4
...
fixed the behaviour Multi paralle
2025-02-20 00:11:37 +01:00
Yannick Stephan
80a61d7e7a
fixed the behaviour
2025-02-20 00:09:46 +01:00
Yannick Stephan
f52c53b41c
Merge pull request #889 from YanSte/parelle-3
...
Multi batches
2025-02-19 23:55:59 +01:00
Yannick Stephan
efdc8a2d26
multi batches
2025-02-19 23:53:25 +01:00
Yannick Stephan
64e6765c9a
Merge pull request #888 from YanSte/parelle-2
...
Parelle
2025-02-19 23:47:36 +01:00
Yannick Stephan
4875283d9f
make it smalled
2025-02-19 23:45:51 +01:00
Yannick Stephan
0567a68c31
Merge pull request #887 from YanSte/paralle
...
Improved Paralles
2025-02-19 23:32:57 +01:00
