4716 Commits

Author SHA1 Message Date
yangdx
eded6d1187 Unify document chunks context format in only_need_context query
- Update Document Chunks label to include (DC) abbreviation
2025-08-08 00:02:53 +08:00
Matt23-star
727ca43d3c feat: add vector index creation functionality for PostgreSQL 2025-08-07 23:07:18 +08:00
yangdx
7780776af6 Update env.example 2025-08-06 18:50:58 +08:00
Daniel.y
a6ef29cef6
Merge pull request #1915 from danielaskdd/optimize-llm-cache
Refact: Optimized LLM Cache Hash Key Generation by Including All Query Parameters
2025-08-06 01:04:02 +08:00
yangdx
2dab4e321d Bump api version to 0199 2025-08-06 01:03:35 +08:00
yangdx
a04c11a598 Remove deprecated storage 2025-08-06 00:02:50 +08:00
yangdx
c22315ea6d refactor: remove selective LLM cache clearing functionality
- Remove optional 'modes' parameter from aclear_cache() and clear_cache() methods
- Replace deprecated drop_cache_by_modes() with drop() method for complete cache clearing
- Update API endpoint to ignore mode-specific parameters and clear all cache
- Simplify frontend clearCache() function to send empty request body

This change ensures all LLM cache is cleared together.
2025-08-05 23:51:51 +08:00
yangdx
cc1f7118e7 Remove deprecated cache_by_modes functionality from all storage 2025-08-05 23:20:26 +08:00
yangdx
8294d6d1b7 Remove deprecated mode field from LLM cache schema
- Drop mode column from LLM cache table
- Update primary key to exclude mode
- Remove mode from all SQL queries
- Deprecate mode-related methods
- Update schema migration logic
2025-08-05 23:18:54 +08:00
yangdx
0b5c708660 Update storage implementation documentation
- Add detailed storage type descriptions
- Remove Chroma from vector storage options
- Include recommended PostgreSQL version
- Add Memgraph to graph storage options
- Update performance comparison notes
2025-08-05 18:03:51 +08:00
yangdx
0463963520 fix: include all query parameters in LLM cache hash key generation
- Add missing query parameters (top_k, enable_rerank, max_tokens, etc.) to cache key generation in kg_query, naive_query, and extract_keywords_only functions
- Add queryparam field to CacheData structure and PostgreSQL storage for debugging
- Update PostgreSQL schema with automatic migration for queryparam JSONB column
- Prevent incorrect cache hits between queries with different parameters

Fixes issue where different query parameters incorrectly shared the same cached results.
2025-08-05 18:03:10 +08:00
yangdx
cb75e6631e Remove quantized embedding info from LLM cache
- Delete quantize_embedding function
- Delete dequantize_embedding function
- Remove embedding fields from CacheData
- Update save_to_cache to exclude embedding data
- Clean up unused quantization-related code
2025-08-05 17:58:34 +08:00
Daniel.y
c7d17f13c1
Merge pull request #1914 from danielaskdd/feat-tiktoken-cache
feat: add tiktoken cache directory support for offline deployment
2025-08-05 14:25:10 +08:00
yangdx
8dd3069005 feat: add tiktoken cache directory support for offline deployment
- Mount tiktoken cache volume in docker-compose
- Add TIKTOKEN_CACHE_DIR environment variable
- Update env.example with tiktoken cache configuration
- Added /data to gitignore
- Added tiktoken env example
2025-08-05 14:22:19 +08:00
yangdx
01bce8c26e feat: add warning logs for deleting non-completed documents 2025-08-05 12:21:08 +08:00
Daniel.y
793e82ae89
Merge pull request #1913 from danielaskdd/fix-base64-for-embedding
Feat: Change embedding formats from float to base64 for efficiency
2025-08-05 11:48:53 +08:00
yangdx
6ff25210ea feat: improve Jina API error handling to show clean messages instead of HTML 2025-08-05 11:46:02 +08:00
yangdx
c5babf61d7 Feat: Change embedding formats from float to base64 for efficiency
- Add base64 support for Jina embeddings
- Add base64 support for OpenAI embeddings
- Update env.example with new embedding options
2025-08-05 11:38:40 +08:00
yangdx
4d492abf41 feat: implement temperature priority cascade for LLM bindings
- Add global --temperature command line argument with env fallback
- Implement temperature priority for Ollama LLM binding:
  1. --ollama-llm-temperature (highest)
  2. OLLAMA_LLM_TEMPERATURE env var
  3. --temperature command arg
  4. TEMPERATURE env var (lowest)
- Implement same priority logic for OpenAI/Azure OpenAI LLM binding
- Ensure command line args always override environment variables
- Maintain backward compatibility with existing configurations
2025-08-05 04:53:55 +08:00
Daniel.y
51deee3d82
Merge pull request #1910 from danielaskdd/openai-option
feat: Add OpenAI LLM Options Support
2025-08-05 03:55:23 +08:00
yangdx
adf7ec8e35 feat: Add OpenAI LLM Options support with BindingOptions framework
- Add OpenAILLMOptions dataclass with full OpenAI API parameter support
- Integrate OpenAI options in config.py for automatic binding detection
- Update server functions to inject OpenAI options for openai/azure_openai bindings
2025-08-05 03:47:26 +08:00
yangdx
3099748668 Add temperature fallback for Ollama LLM binding
- Implement OLLAMA_LLM_TEMPERATURE env var
- Fallback to global TEMPERATURE if unset
- Remove redundant OllamaLLMOptions logic
- Update env.example with new setting
2025-08-05 01:50:09 +08:00
Daniel.y
936809a084
Merge pull request #1909 from danielaskdd/fix-ollama
Fix ollama stop option handling and enhance temperature configuration
2025-08-04 23:51:29 +08:00
yangdx
e5e3f0f878 Fix(Ollama option): change stop option from string to list and add fallback global temperature setting 2025-08-04 19:43:14 +08:00
yangdx
f8a880ac66 Improved binding options testing and documentation 2025-08-04 18:21:55 +08:00
yangdx
7b3a9c09ca Fix: add missing colume to LLM cache of PostgreSQL implementation 2025-08-04 11:12:59 +08:00
yangdx
223f4cdf62 Mark deprecated fields with TODO comments 2025-08-04 11:11:57 +08:00
yangdx
3bac19df5a Bump core version to 1.4.7 and api version to 0198 2025-08-04 10:55:41 +08:00
yangdx
63496698a1 Fix: ensure data migration is handled by single-process
- Wrap migration logic with get_data_init_lock() to ensure single-process execution
- Prevent race conditions when multiple processes start simultaneously
v1.4.6
2025-08-04 01:47:20 +08:00
yangdx
e04d8ed8a7 Improved storage drop logging with namespace details
- Added namespace and workspace to drop logs
2025-08-04 00:56:39 +08:00
Daniel.y
7de0276a57
Merge pull request #1904 from HKUDS/optimize-doc-delete
feat(performance): Optimize Document Deletion with Entity/Relation Indexing
2025-08-04 00:39:02 +08:00
yangdx
5513155808 Fix namespace tablename translate error
- Reorder namespace table map for PostgreSQL
- Ensure specific namespaces come first
2025-08-04 00:21:20 +08:00
yangdx
daf2633dc2 Bump api version to 0198 2025-08-03 23:04:42 +08:00
yangdx
2f1d1b69e5 Bump api version to 0197 2025-08-03 23:03:48 +08:00
yangdx
7505195303 fix: add full_entities and full_relations to clear_documents storage list 2025-08-03 23:02:58 +08:00
yangdx
952d1feb07 feat: Add support for KV_STORE_FULL_ENTITIES and KV_STORE_FULL_RELATIONS namespaces in PGKVStorage
- Add LIGHTRAG_FULL_ENTITIES and LIGHTRAG_FULL_RELATIONS table schemas
- Implement complete CRUD operations for both namespaces
- Add automatic table creation and migration support
- Add SQL templates and namespace mappings
- Ensure workspace isolation and proper indexing
2025-08-03 22:54:56 +08:00
yangdx
bf9a6d699b Fix(lightrag): Handle undirected edges in data migration
The `_migrate_entity_relation_data` function previously processed directed edges from `get_all_edges`, which could lead to duplicates (e.g., (A,B) and (B,A)) and an incorrect relation count.

This commit normalizes edges by sorting their source and target nodes before adding them to the relation set. This ensures all edges are treated as undirected and are properly deduplicated.
2025-08-03 22:14:24 +08:00
yangdx
e8d8afa846 Removed auto storage management from LightRAG instance creation
- The `initialize_storages` method must be explicitly called after LightRAG creation.
The `finalize_storages` method should be called before LightRAG destyoyed.
- Added explicit data migration check
2025-08-03 12:42:57 +08:00
yangdx
06efab4af2 Revert "Remove auto_manage_storages_states option"
This reverts commit bfe6657b316f7e50bc9c5f0cc71d9fbb2b605ddd.
2025-08-03 12:12:13 +08:00
yangdx
d2dd137f83 feat: implement get_all_nodes and get_all_edges methods for graph storage backends
Add get_all_nodes() and get_all_edges() methods to Neo4JStorage, PGGraphStorage, MongoGraphStorage, and MemgraphStorage classes. These methods return all nodes and edges in the graph with consistent formatting matching NetworkXStorage for compatibility across different storage backends.
2025-08-03 11:02:37 +08:00
yangdx
bfe6657b31 Remove auto_manage_storages_states option
- Always manage storage states by LightRAG
- Remove rag.initialize_storages() from all examples
2025-08-03 10:29:36 +08:00
yangdx
091f2b42c3 feat(performance): Optimize document deletion with entity/relation index
- Introduces an index mapping documents to their corresponding entities and relations. This significantly speeds up `adelete_by_doc_id` by replacing slow graph traversal with a fast key-value lookup.
- Refactors the ingestion pipeline (`merge_nodes_and_edges`) to populate this new index. Adds a one-time data migration script to backfill the index for existing data.
2025-08-03 09:19:02 +08:00
yangdx
2f0aa7ed12 Optimize graph query by simplifying MATCH pattern
- Simplify MATCH clause to ()-[r]-()
- Remove node type constraints
- Improve query performance
2025-08-02 12:54:22 +08:00
Daniel.y
a417f7c168
Merge pull request #1899 from danielaskdd/kv-storage-workspace
Fix: workspace isolation problem for json KV storage
2025-08-02 11:36:20 +08:00
yangdx
e00690b41b Fix: workspace isolation problem for json KV storage
- Use workspace+namespace as final_namespace identifier
- Update all related storage operations
- Maintain backward compatibility
2025-08-02 11:30:19 +08:00
Daniel.y
f6b90fe482
Merge pull request #1897 from danielaskdd/json-repair
refactor: improve JSON parsing reliability with json-repair library
2025-08-01 20:15:44 +08:00
yangdx
d98fe6f340 Move json-repair to main dependencies 2025-08-01 19:59:39 +08:00
yangdx
32af45ff46 refactor: improve JSON parsing reliability with json-repair library
Replace regex-based JSON extraction with json-repair for better handling of malformed LLM responses. Remove deprecated JSON parsing utilities and clean up keyword_extraction parameter across LLM providers.

- Remove locate_json_string_body_from_string() and convert_response_to_json()
- Use json-repair.loads() in extract_keywords_only() for robust parsing
- Clean up LLM interfaces and remove unused parameters
- Add json-repair dependency
2025-08-01 19:36:20 +08:00
Daniel.y
3ae9c63b26
Merge pull request #1895 from danielaskdd/finalize-on-exit
Add graceful shutdown handling for LightRAG server
2025-08-01 12:12:40 +08:00
yangdx
fdf051c234 Add graceful shutdown handling for LightRAG server
- Setup signal handlers for SIGINT/SIGTERM
- Clean up shared resources on shutdown
- Finalize shared data storage
2025-08-01 10:56:18 +08:00