yangdx
adb4ca9294
Fix linting
2025-03-28 16:49:35 +08:00
yangdx
35d4503677
Feat: add duplicate file check
2025-03-28 16:48:46 +08:00
yangdx
d456269718
feat: Add file name display in WebUI
...
Backend:
- Add file_path field to DocStatusResponse
- Update document status response creation
Frontend:
- Remove metadata column
- Improve filename display with truncation and tooltips
- Add show/hide filename toggle with proper styling
- Update translations for all supported languages"
2025-03-25 22:42:46 +08:00
yangdx
35df22cf08
Update pipeline_enqueue_file to include file_paths parameter
2025-03-25 20:47:25 +08:00
yangdx
d05cf286f4
Fix linting
2025-03-24 14:30:17 +08:00
yangdx
9e3994419e
Refactor authentication logic and Swagger UI config
...
- Consolidate authentication dependencies
- Improve Swagger UI security parameters
2025-03-24 14:29:36 +08:00
yangdx
79bf26dfeb
Process update flags status for proper boolean conversion
...
- Convert MutableBoolean to regular boolean values
- Handle both multiprocess and single process cases
2025-03-24 12:49:39 +08:00
yangdx
77ab26e5bf
Fix linting
2025-03-24 05:39:50 +08:00
yangdx
071302d10f
Moved update status logic to document routes.
...
- Removed update status from health check endpoint
- Added update_status field to PipelineStatusResponse
2025-03-24 05:37:03 +08:00
yangdx
90ef55960d
Refactor authentication and whitelist handling
...
- Combined auth and API key dependencies
- Optimized whitelist path matching
- Added optional API key to OllamaAPI
2025-03-24 05:23:40 +08:00
yangdx
7e8a2c0e9b
Added temperature parameter for LLM
2025-03-24 02:02:34 +08:00
yangdx
53396e4d82
Fixlinting
2025-03-21 16:56:47 +08:00
yangdx
0761af19c6
Files are now processed in batches in auto scan
2025-03-21 13:41:37 +08:00
yangdx
168232803b
Optimized the document processing pipeline
...
- Enqueue all files to pipeline before starting process when scanning input folder
- Changed parallel enqueue to sequential to prevent CPU overload by heavy file extraction jobs
2025-03-21 13:08:57 +08:00
yangdx
73440e3a96
Fix linting
2025-03-12 16:28:55 +08:00
yangdx
16e06100e7
Add PipelineStatusResponse model for API endpoint
2025-03-12 16:16:39 +08:00
Saifeddine ALOUI
53cfb72db4
linted
2025-03-08 23:07:51 +01:00
zrguo
c67e6aceca
Merge branch 'main' into main
2025-03-09 00:23:06 +08:00
Milin
c015296081
refactor(api): Fix issues reported by pre-commit
...
- Modified code layout and formatting in multiple files, improving code readability.
- Updated import statements, removing unused libraries.
- Simplified the writing of some functions and exception handling.
2025-03-06 14:23:52 +08:00
Saifeddine ALOUI
00f3c6c6dd
Upgraded document loading engine
2025-03-06 01:11:48 +01:00
Saifeddine ALOUI
6e4daea056
Linting
2025-03-05 15:36:47 +01:00
Saifeddine ALOUI
aaa8194423
Update document_routes.py
2025-03-05 15:32:39 +01:00
Milin
852634e20e
Merge branch 'main' into feat_login-jwt
2025-03-05 12:42:40 +08:00
Milin
63aa4f9dfc
feat(api): Add user authentication functionality
...
- Implement JWT-based user authentication logic
- Add login endpoint and token validation middleware
- Update API routes with authentication dependencies
- Add authentication-related environment variables
- Optimize requirements.txt with necessary dependencies
2025-03-05 11:09:31 +08:00
zrguo
9d56b1c8a4
Merge pull request #976 from danielaskdd/validate-content-before-enqueue
...
Improved file handling and validation for document processing
2025-03-04 13:26:54 +08:00
yangdx
61839f311a
Fix package name checks for docx and pptx modules.
...
- Added type ignore for package checks
- Corrected docx pptx package name for new version
2025-03-04 10:00:07 +08:00
yangdx
aa5888042e
Improved file handling and validation for document processing
...
• Enhanced UTF-8 validation for text files
• Added content validation checks
• Better handling of binary data
• Added logging for ignored document IDs
• Improved document ID filtering
2025-03-02 23:57:57 +08:00
yangdx
d704512139
Refactor shared storage module to improve async handling and naming consistency
...
• Add async support for get_namespace_data
• Rename get_update_flags to get_update_flag
• Rename set_update_flag to set_all_update_flags
• Update docstrings for clarity
• Fix typos in log messages
2025-03-01 05:01:26 +08:00
yangdx
c973498c34
Fix linting
2025-02-28 21:35:04 +08:00
yangdx
157ec862ae
Enhance logging system with file rotation and unified configuration
...
• Unify logging across Gunicorn and Uvicorn
• Add rotating file handlers
2025-02-28 14:57:25 +08:00
yangdx
8cd45161f2
feat: add history_messages to track pipeline processing progress
...
• Add shared history_messages list
• Track pipeline progress with messages
2025-02-28 13:53:40 +08:00
yangdx
04bd5413c9
Add API endpoint to retrieve document indexing pipeline status
...
• GET /pipeline_status endpoint added
• Returns current pipeline processing state
2025-02-28 12:21:50 +08:00
yangdx
b4bcd76599
Remove useless scan progress tracking functionality and related code
2025-02-28 10:53:36 +08:00
yangdx
db2a902fcb
Rename get_scan_lock to get_storage_lock
2025-02-28 00:34:33 +08:00
yangdx
92ecb0da97
Refactor document scanning progress share variable initialization
2025-02-27 16:07:00 +08:00
yangdx
7436c06f6c
Fix linting
2025-02-26 18:11:16 +08:00
yangdx
4eb069d1d6
Initialize scan_progress with default values if not already set
2025-02-26 17:42:49 +08:00
yangdx
2752a764ae
Refactor storage implementations to support both single and multi-process modes
...
• Add shared storage management module
• Support process/thread lock based on mode
2025-02-26 05:38:38 +08:00
yangdx
d74a23d2cc
Add multiple workers support for API Server
2025-02-25 09:37:00 +08:00
yangdx
dbeda8a9ff
Change scanning logs from INFO to DEBUG level
2025-02-23 16:12:08 +08:00
yangdx
b144e0c3b0
Sync modifications from main branch
2025-02-21 21:07:37 +08:00
yangdx
1749678384
Fix linting
2025-02-21 12:16:04 +08:00
yangdx
f52b9929bb
fix: add missing rag parameter in pipeline_enqueue_file call
...
• Add rag param to function call
• Fix argument mismatch error
• Ensure proper pipeline execution
2025-02-20 14:48:54 +08:00
yangdx
82a4cb3e79
Fix refactoring error on document handling
...
- Fix refactoring error on pipeline_index_file
- Delete unsed func: scan_directory
- Add type hints of rag for better maintainability
- Refine comments for better understanding
2025-02-20 14:30:41 +08:00
yangdx
a8abcf14ac
Fit linting
2025-02-20 04:12:21 +08:00
yangdx
c0c87edc45
split lightrag_servery.py to smaller files
2025-02-20 03:26:39 +08:00
