yujunjun/LightRAG
1
0
Fork 0
mirror of https://github.com/HKUDS/LightRAG.git synced 2025-07-08 17:43:20 +00:00
Code Issues Packages Projects Releases Wiki Activity
1,210 Commits 4 Branches 40 Tags
Commit Graph

99 Commits

Author SHA1 Message Date
hyb
3dba406644 feat: Added webui management, including file upload, text upload, Q&A query, graph database management (can view tags, view knowledge graph based on tags), system status (whether it is good, data storage status, model status, path),request /webui/index.html 2025-01-25 18:38:46 +08:00
Saifeddine ALOUI
34018cb1e0 Separated llms from the main llm.py file and fixed some deprication bugs 2025-01-25 00:11:00 +01:00
zrguo
cdf967cedd
Merge pull request #631 from 18277486571HYB/redis_impl 
feat: 新增ini文件读取数据库配置方式，方便生产环境，修改Lightrag ainsert方法_add_doc_keys获取方式，原…
 2025-01-25 01:44:46 +08:00
Magic_yuan
443aab2882 修复当出现异常时，会导致更新数据卡死的bug 2025-01-24 10:15:25 +08:00
hyb
ff71952c8c feat: 新增ini文件读取数据库配置方式，方便生产环境，修改Lightrag ainsert方法_add_doc_keys获取方式，原来只过滤存在的，但这会让失败的文档无法再次存储，新增--chunk_size和--chunk_overlap_size方便生产环境，新增llm_binding：openai-ollama 方便用openai的同时使用ollama embedding 2025-01-23 22:58:57 +08:00
hyb
e08905b398 feat: 增加redis KV存储，增加openai+neo4j+milvus+redis的demo测试，新增lightrag.py: RedisKVStorage，新增requirements.txt:aioredis依赖 2025-01-22 16:42:13 +08:00
zrguo
b84aab5cd0
Merge pull request #590 from jin38324/main 
Enhance Robustness of insert Method with Pipeline Processing and Caching Mechanisms
 2025-01-16 14:20:08 +08:00
jin
6ae8647285 support pipeline mode 2025-01-16 12:58:15 +08:00
jin
d5ae6669ea support pipeline mode 2025-01-16 12:52:37 +08:00
jin
17a2ec2bc4
Merge branch 'HKUDS:main' into main 2025-01-16 09:59:27 +08:00
Gurjot Singh
bc79f6650e Fix linting errors 2025-01-14 22:23:14 +05:30
Gurjot Singh
ef61ffe444 Add custom function with separate keyword extraction for user's query and a separate prompt 2025-01-14 22:10:47 +05:30
jin
85331e3fa2 update Oracle support 
add cache support, fix bug
 2025-01-10 11:36:28 +08:00
adikalra
acde4ed173 Add custom chunking function. 2025-01-09 17:20:24 +05:30
zrguo
b93203804c
Merge branch 'main' into main 2025-01-09 15:28:57 +08:00
zrguo
92ccfa2770
Merge pull request #555 from ParisNeo/main 
Restore backwards compatibility for LightRAG's ainsert method
 2025-01-09 15:27:09 +08:00
童石渊
dd213c95be 增加仅字符分割参数，如果开启，仅采用字符分割，不开启，在分割完以后如果chunk过大，会继续根据token size分割，更新测试文件 2025-01-09 11:55:49 +08:00
Saifeddine ALOUI
65c1450c66 fixed retro compatibility with ainsert by making split_by_character get a None default value 2025-01-08 20:50:22 +01:00
Gurjot Singh
9565a4663a Fix trailing whitespace and formatting issues in lightrag.py 2025-01-09 00:39:22 +05:30
Gurjot Singh
a940251390 Implement custom chunking feature 2025-01-07 20:57:39 +05:30
童石渊
6b19401dc6 chunk split retry 2025-01-07 16:26:12 +08:00
童石渊
536d6f2283 添加字符分割功能，在“insert”函数中如果增加参数split_by_character，则会按照split_by_character进行字符分割，此时如果每个分割后的chunk的tokens大于max_token_size，则会继续按token_size分割（todo：考虑字符分割后过短的chunk处理） 2025-01-07 00:28:15 +08:00
zrguo
990b684a85 Update lightrag.py 2025-01-06 15:27:31 +08:00
Samuel Chan
6ae27d8f06 Some enhancements: 
- Enable the llm_cache storage to support get_by_mode_and_id, to improve the performance for using real KV server
- Provide an option for the developers to cache the LLM response when extracting entities for a document. Solving the paint point that sometimes the process failed, the processed chunks we need to call LLM again, money and time wasted. With the new option (by default not enabled) enabling, we can cache that result, can significantly save the time and money for beginners.
 2025-01-06 12:50:05 +08:00
Samuel Chan
60e8a355f0
Merge branch 'HKUDS:main' into main 2025-01-03 21:18:17 +08:00
Samuel Chan
b17cb2aa95 With a draft for progres_impl 2025-01-01 22:43:59 +08:00
zrguo
d489d9dec0 fix linting errors 2024-12-31 17:32:04 +08:00
zrguo
cee5b2fbb0 add delete by doc id 2024-12-31 17:15:57 +08:00
Magic_yuan
aaaf617451 feat(lightrag): Implement mix search mode combining knowledge graph and vector retrieval 
- Add 'mix' mode to QueryParam for hybrid search functionality
- Implement mix_kg_vector_query to combine knowledge graph and vector search results
- Update LightRAG class to handle 'mix' mode queries
- Enhance README with examples and explanations for the new mix search mode
- Introduce new prompt structure for generating responses based on combined search results
 2024-12-28 11:56:28 +08:00
Magic_yuan
650b8e38b7 feat(lightrag): Add document status tracking and checkpoint support 
功能(lightrag): 添加文档状态跟踪和断点续传支持

- Add DocStatus enum and DocProcessingStatus class for document processing state management
- 添加 DocStatus 枚举和 DocProcessingStatus 类用于文档处理状态管理

- Implement JsonDocStatusStorage for persistent status storage
- 实现 JsonDocStatusStorage 用于持久化状态存储

- Add document-level deduplication in batch processing
- 在批处理中添加文档级别的去重功能

- Add checkpoint support in ainsert method for resumable document processing
- 在 ainsert 方法中添加断点续传支持，实现可恢复的文档处理

- Add status query methods for monitoring processing progress
- 添加状态查询方法用于监控处理进度

- Update LightRAG initialization to support document status tracking
- 更新 LightRAG 初始化以支持文档状态跟踪
 2024-12-28 00:11:25 +08:00
zrguo
457e683acd
Update lightrag.py 2024-12-26 22:14:04 +08:00
Alex Potapenko
6f71293c83 Add Gremlin graph storage 2024-12-19 17:47:42 +01:00
Weaxs
344d8f277b support TiDBGraphStorage 2024-12-18 10:57:33 +08:00
GG
2d048b5eb0 fix(llm): hashing_kv初始化修复 
-hybrid模式对hashing_kv的依赖不止global_config，干脆复用llm_response_cache的初始化结构
 2024-12-17 16:44:42 +08:00
Alex Potapenko
7564841450 Add Apache AGE graph storage 2024-12-13 20:41:38 +01:00
Weaxs
288985eab4 pre-commit fix tidb 2024-12-12 10:22:31 +08:00
Weaxs
8ef5a6b8cd support TiDB: add TiDBKVStorage, TiDBVectorDBStorage 2024-12-11 16:23:50 +08:00
zrguo
504a3c233b
Merge branch 'main' into pkaushal/vectordb-chroma 2024-12-11 14:21:36 +08:00
Pankaj Kaushal
ca788463cc feat: Add ChromaDB integration for vector storage 
- Implemented `ChromaVectorDBStorage` class in `lightrag/kg/chroma_impl.py` to support ChromaDB as a vector storage backend.
- Updated `lightrag.py` to include `ChromaVectorDBStorage` in the storage class mapping.
- Added a test script `test_chromadb.py` to demonstrate the usage of ChromaDB with LightRAG, including configuration for embedding functions and ChromaDB connection settings.
- fix lazy import function to support package context for dynamic class loading.
  288d4b8355
 2024-12-10 16:23:05 +01:00
david
288d4b8355 fix lazy import 2024-12-10 17:16:21 +08:00
zrguo
3e112c0d05
Merge pull request #432 from ChenZiHong-Gavin/main 
fix(lightrag): use is_closed() instead of _closed
 2024-12-09 18:08:43 +08:00
zrguo
4c89a1a620
Merge pull request #429 from davidleon/improvement/lazy_external_load 
fix extra kwargs error: keyword_extraction.
 2024-12-09 18:07:30 +08:00
chenzihong
9dd51f1f35 fix(lightrag): use is_closed() instead of _closed 2024-12-09 17:10:13 +08:00
david
9717ad87fc fix extra kwargs error: keyword_extraction. 
add lazy_external_load to reduce external lib deps whenever it's not necessary for user.
 2024-12-09 15:35:35 +08:00
Magic_yuan
ccf44dc334 feat(cache): 增加 LLM 相似性检查功能并优化缓存机制 
- 在 embedding 缓存配置中添加 use_llm_check 参数
- 实现 LLM 相似性检查逻辑，作为缓存命中的二次验证- 优化 naive 模式的缓存处理流程
- 调整缓存数据结构，移除不必要的 model 字段
 2024-12-08 17:35:52 +08:00
magicyuan876
d48c6e4588 feat(lightrag): 添加 查询时使用embedding缓存功能 
- 在 LightRAG 类中添加 embedding_cache_config配置项
- 实现基于 embedding 相似度的缓存查询和存储
- 添加量化和反量化函数，用于压缩 embedding 数据
- 新增示例演示 embedding 缓存的使用
 2024-12-06 08:17:20 +08:00
partoneplay
d8ba7c57f3 Add MongoDB as KV storage 2024-12-05 13:57:43 +08:00
zrguo
6d274019dd Merge pull request #393 from partoneplay/main 
Add Milvus as vector storage
 2024-12-05 12:05:30 +08:00
partoneplay
052322b213 Add Milvus as vector storage 2024-12-05 08:48:41 +08:00
LarFii
44d441a951 update insert custom kg 2024-12-04 19:44:04 +08:00