1041 Commits

Author SHA1 Message Date
yangdx
62cdc7d7eb Update documentation with LLM selection guidelines and API improvements 2025-08-21 13:59:14 +08:00
yangdx
0e67ead8fa Rename MAX_TOKENS to SUMMARY_MAX_TOKENS for clarity 2025-08-21 10:15:20 +08:00
yangdx
aa22772721 Refactor LLM temperature handling to be provider-specific
• Remove global temperature parameter
• Add provider-specific temp configs
• Update env example with new settings
• Fix Bedrock temperature handling
• Clean up splash screen display
2025-08-20 23:52:33 +08:00
yangdx
df7bcb1e3d Add LLM_TIMEOUT configuration for all LLM providers
- Add LLM_TIMEOUT env variable
- Apply timeout to all LLM bindings
2025-08-20 23:50:57 +08:00
yangdx
4c556d8aae Set default TIMEOUT value to 150, and gunicorn timeout to TIMEOUT+30 2025-08-20 22:04:32 +08:00
yangdx
9b7ed84e05 Improve document deletion error handling and message consistency
- Standardize deletion log messages
- Add try-catch for file operations
- Improve enqueued file error handling
2025-08-20 11:01:24 +08:00
yangdx
2603e99005 Enhance file deletion to remove files from both input and enqueued dirs 2025-08-19 17:13:58 +08:00
yangdx
1f86543772 Update i18n translation and webui assets 2025-08-19 16:23:05 +08:00
yangdx
0aa1bc8bf9 Update webui assets and bump api version to 0205 2025-08-19 15:11:34 +08:00
yangdx
9ed5b93467 Add [File Extraction] prefix to error messages and logs 2025-08-19 11:33:28 +08:00
yangdx
368d2b00d6 Update webui assets and bump api version to 0204 2025-08-18 19:33:46 +08:00
yangdx
377f1a022e fix: reset PROCESSING/FAILED docs to PENDING at the beginging of document processing pipeline
- Reset documents with PROCESSING/FAILED status to PENDING when they pass consistency checks
- Update doc_status storage and clear error messages/metadata on reset
2025-08-18 00:49:52 +08:00
yangdx
add8b07a21 Improve logging messages for document processing clarity 2025-08-18 00:22:04 +08:00
yangdx
14e083a1a6 fix: replace pyuca with pypinyin for Chinese pinyin sorting and add file_path sort 2025-08-17 15:21:24 +08:00
yangdx
61469c0a56 Add Chinese pinyin sorting support across document operations
• Replace pyuca with centralized utils function
• Add pinyin sort keys for file paths
• Update MongoDB indexes with zh collation
• Migrate existing indexes for compatibility
• Support Chinese chars in Redis/JSON storage
• Keep PostgreSQL sorting order controled by Database Collate order
2025-08-17 12:45:48 +08:00
yangdx
6196bab00a Update webui assets and bump api version to 0203 2025-08-17 10:39:16 +08:00
yangdx
3a7310873c Merge branch 'bedrock-support' 2025-08-17 02:23:44 +08:00
yangdx
da7e4b79e5 Update documentation in README files 2025-08-17 02:23:14 +08:00
yangdx
1ed77a2e53 Remove openai-ollama binding from LightRAG level args 2025-08-17 02:13:50 +08:00
yangdx
301acfc274 Update webui assets 2025-08-17 01:54:39 +08:00
yangdx
45365ff6ef Bump api version to 0202 2025-08-16 23:53:01 +08:00
yangdx
cceb46b320 fix: subdirectories are no longer processed during file scans
• Change rglob to glob for file scanning
• Simplify error logging messages
2025-08-16 23:46:33 +08:00
yangdx
f5b0c3d38c feat: Recording file extraction error status to document pipeline
- Add apipeline_enqueue_error_documents function to LightRAG class for recording file processing errors in doc_status storage
- Enhance pipeline_enqueue_file with detailed error handling for all file processing stages:
  * File access errors (permissions, not found)
  * UTF-8 encoding errors
  * Format-specific processing errors (PDF, DOCX, PPTX, XLSX)
  * Content validation errors
  * Unsupported file type errors

This implementation ensures all file extraction failures are properly tracked and recorded in the doc_status storage system, providing better visibility into document processing issues and enabling improved error monitoring and debugging capabilities.
2025-08-16 23:08:52 +08:00
yangdx
5d00c4c7a8 feat: move processed files to __enqueued__ directory after processing with filename conflicts handling 2025-08-16 13:19:20 +08:00
SJ
f7ca9ae16a Ruff formatted 2025-08-15 22:21:34 +00:00
yangdx
dc7a6e1c5b Update README 2025-08-16 06:15:27 +08:00
SJ
3aa3332505
Merge pull request #1 from HKUDS/main
merge
2025-08-15 17:09:03 -05:00
yangdx
3bba5fc506 Fix linting 2025-08-14 13:03:23 +08:00
yangdx
772f981e7e fix: check and process queued docs even when upload directory is empty 2025-08-14 12:35:39 +08:00
yangdx
fd0ae4646f Fixes crash when processing files with UTF-8 encoding error
- Fix TypeError "cannot unpack non-iterable bool object" in document processing
- Change all error returns from `False` to `(False, "")` for consistency
- Ensure pipeline_enqueue_file always returns tuple (bool, str)
- Add missing return statement for no-content-extracted case
- Improve error handling for UTF-8 encoding issues and unsupported file types
2025-08-14 05:31:38 +08:00
yangdx
3ccd10f1e4 Update webui assets 2025-08-14 05:03:43 +08:00
yangdx
160a40dc04 Bump api version to 0201 2025-08-14 05:02:20 +08:00
yangdx
ae517181ad Bump api version to 0200 2025-08-14 05:01:13 +08:00
SJ
99643f01de
Enhancement: support aws bedrock as an LLm binding #1733 2025-08-13 02:08:13 -05:00
yangdx
2dab4e321d Bump api version to 0199 2025-08-06 01:03:35 +08:00
yangdx
c22315ea6d refactor: remove selective LLM cache clearing functionality
- Remove optional 'modes' parameter from aclear_cache() and clear_cache() methods
- Replace deprecated drop_cache_by_modes() with drop() method for complete cache clearing
- Update API endpoint to ignore mode-specific parameters and clear all cache
- Simplify frontend clearCache() function to send empty request body

This change ensures all LLM cache is cleared together.
2025-08-05 23:51:51 +08:00
yangdx
0b5c708660 Update storage implementation documentation
- Add detailed storage type descriptions
- Remove Chroma from vector storage options
- Include recommended PostgreSQL version
- Add Memgraph to graph storage options
- Update performance comparison notes
2025-08-05 18:03:51 +08:00
yangdx
4d492abf41 feat: implement temperature priority cascade for LLM bindings
- Add global --temperature command line argument with env fallback
- Implement temperature priority for Ollama LLM binding:
  1. --ollama-llm-temperature (highest)
  2. OLLAMA_LLM_TEMPERATURE env var
  3. --temperature command arg
  4. TEMPERATURE env var (lowest)
- Implement same priority logic for OpenAI/Azure OpenAI LLM binding
- Ensure command line args always override environment variables
- Maintain backward compatibility with existing configurations
2025-08-05 04:53:55 +08:00
yangdx
adf7ec8e35 feat: Add OpenAI LLM Options support with BindingOptions framework
- Add OpenAILLMOptions dataclass with full OpenAI API parameter support
- Integrate OpenAI options in config.py for automatic binding detection
- Update server functions to inject OpenAI options for openai/azure_openai bindings
2025-08-05 03:47:26 +08:00
yangdx
3099748668 Add temperature fallback for Ollama LLM binding
- Implement OLLAMA_LLM_TEMPERATURE env var
- Fallback to global TEMPERATURE if unset
- Remove redundant OllamaLLMOptions logic
- Update env.example with new setting
2025-08-05 01:50:09 +08:00
yangdx
63496698a1 Fix: ensure data migration is handled by single-process
- Wrap migration logic with get_data_init_lock() to ensure single-process execution
- Prevent race conditions when multiple processes start simultaneously
2025-08-04 01:47:20 +08:00
yangdx
e04d8ed8a7 Improved storage drop logging with namespace details
- Added namespace and workspace to drop logs
2025-08-04 00:56:39 +08:00
yangdx
daf2633dc2 Bump api version to 0198 2025-08-03 23:04:42 +08:00
yangdx
2f1d1b69e5 Bump api version to 0197 2025-08-03 23:03:48 +08:00
yangdx
7505195303 fix: add full_entities and full_relations to clear_documents storage list 2025-08-03 23:02:58 +08:00
yangdx
e8d8afa846 Removed auto storage management from LightRAG instance creation
- The `initialize_storages` method must be explicitly called after LightRAG creation.
The `finalize_storages` method should be called before LightRAG destyoyed.
- Added explicit data migration check
2025-08-03 12:42:57 +08:00
yangdx
06efab4af2 Revert "Remove auto_manage_storages_states option"
This reverts commit bfe6657b316f7e50bc9c5f0cc71d9fbb2b605ddd.
2025-08-03 12:12:13 +08:00
yangdx
bfe6657b31 Remove auto_manage_storages_states option
- Always manage storage states by LightRAG
- Remove rag.initialize_storages() from all examples
2025-08-03 10:29:36 +08:00
yangdx
fdf051c234 Add graceful shutdown handling for LightRAG server
- Setup signal handlers for SIGINT/SIGTERM
- Clean up shared resources on shutdown
- Finalize shared data storage
2025-08-01 10:56:18 +08:00
yangdx
0f624b594a Update webui assets 2025-07-31 23:53:23 +08:00