### What problem does this PR solve?
This is a cherry-pick from #7781 as requested.
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
Co-authored-by: Kevin Hu <kevinhu.sh@gmail.com>
### What problem does this PR solve?
Try the best to repair corrupted PDF files on upload automatically.
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
### What problem does this PR solve?
Hello, our use case requires LLM agent to invoke some tools, so I made a
simple implementation here.
This PR does two things:
1. A simple plugin mechanism based on `pluginlib`:
This mechanism lives in the `plugin` directory. It will only load
plugins from `plugin/embedded_plugins` for now.
A sample plugin `bad_calculator.py` is placed in
`plugin/embedded_plugins/llm_tools`, it accepts two numbers `a` and `b`,
then give a wrong result `a + b + 100`.
In the future, it can load plugins from external location with little
code change.
Plugins are divided into different types. The only plugin type supported
in this PR is `llm_tools`, which must implement the `LLMToolPlugin`
class in the `plugin/llm_tool_plugin.py`.
More plugin types can be added in the future.
2. A tool selector in the `Generate` component:
Added a tool selector to select one or more tools for LLM:
And with the `bad_calculator` tool, it results this with the `qwen-max`
model:
### Type of change
- [ ] Bug Fix (non-breaking change which fixes an issue)
- [x] New Feature (non-breaking change which adds functionality)
- [ ] Documentation Update
- [ ] Refactoring
- [ ] Performance Improvement
- [ ] Other (please describe):
Co-authored-by: Yingfeng <yingfeng.zhang@gmail.com>
### What problem does this PR solve?
Add MCP support with a client example.
Issue link: #4344
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
### What problem does this PR solve?
This PR addresses the build and dependency issues faced by developers in
regions with poor connectivity to official Ubuntu repositories and
standard dependency sources. Currently, developers in these regions
experience slow or failed Docker builds and dependency downloads,
significantly impacting development efficiency.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [ ] New Feature (non-breaking change which adds functionality)
- [ ] Documentation Update
- [ ] Refactoring
- [ ] Performance Improvement
- [ ] Other (please describe):
The changes include:
1. Modified Dockerfile to use alternative Ubuntu mirrors with better
connectivity in affected regions
2. Added a new script (download_deps_CN.py) that provides
region-specific alternative download links for dependencies
### What problem does this PR solve?
This PR gives better control over how we distribute which service will
be loaded. With this approach, we can create containers to run only the
web server and others to run the task executor. It also introduces the
unique ID per task executor host, this will be important when scaling
task executors horizontally, considering unique task executor ids will
be required.
This new `entrypoint.sh` maintains the default behavior of starting the
web server and task executor in the same host.
### Type of change
- [ ] Bug Fix (non-breaking change which fixes an issue)
- [X] New Feature (non-breaking change which adds functionality)
- [ ] Documentation Update
- [ ] Refactoring
- [ ] Performance Improvement
- [ ] Other (please describe):
### What problem does this PR solve?
Introduced jemalloc.
Python uses pymalloc (which is an reimplementation of gblibc malloc) to
manage RES. It has pools for small objects to avoid returning memory to
OS aggressively. My experience is: Replacing pymalloc with
[jemalloc](https://github.com/jemalloc/jemalloc) can reduce RES and
speedup task_executor.py.
### Type of change
- [x] Performance Improvement
### What problem does this PR solve?
Replaced pypi.tuna.tsinghua.edu.cn with mirrors.aliyun.com/pypi.
I notice aliyun.com sometimes is much faster than tsinghua.edu.
### Type of change
- [x] Refactoring
### What problem does this PR solve?
https://github.com/infiniflow/ragflow/issues/4319
This pull request includes several changes to improve the Docker setup
and documentation for the project. The most important changes include
updating the Dockerfile to support modern versions of Rust, adding a new
Docker Compose configuration for macOS, and updating the build
instructions in the documentation.
Improvements to Docker setup:
*
[`Dockerfile`](diffhunk://#diff-dd2c0eb6ea5cfc6c4bd4eac30934e2d5746747af48fef6da689e85b752f39557L80-R107):
Added installation steps for a modern version of Rust and updated the
logic for installing the correct ODBC driver based on the architecture.
*
[`docker/docker-compose-macos.yml`](diffhunk://#diff-8e8587143bb2442c02f6dff4caa217ebbe3ba4ec8e7c23b2e568886a67b00eafR1-R56):
Added a new Docker Compose configuration file specifically for macOS,
including service dependencies, environment variables, and volume
mappings.
Updates to documentation:
*
[`docs/guides/develop/build_docker_image.mdx`](diffhunk://#diff-d6136bb897f7245aae33b0accbcf7c508ceaef005c545f9f09cad3cada840a19L44-R44):
Updated the build instructions to use the new Docker Compose
configuration for macOS instead of the previous Docker build command.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [x] Documentation Update
---------
Signed-off-by: Samuel Giffard <samuel.giffard@mytomorrows.com>
### What problem does this PR solve?
Change index url per NEED_MIRROR. Close#4507
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
Fix: After executing npm i --force locally, the login page cannot be
opened #4290
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
For the new feature Add mssql support in the Dockerfile, I suggest
including support for msodbcsql18 for ARM64.
Based on current testing results (on macOS ARM64 environment),
msodbcsql18 needs to be installed.
I hope future Dockerfiles can incorporate a conditional check for this.
Specifically:
When $ARCH=arm64 (macOS ARM64 environment), install msodbcsql18.
In other cases (general x86_64 environment), install msodbcsql17.
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
Co-authored-by: Mage Lu <magelu@MagedeMac-mini.local>
some thing
- execsql add connection mssql
- fix bug duckduckgo-search rate limit
- update typo vi res
---------
Co-authored-by: Kevin Hu <kevinhu.sh@gmail.com>
### What problem does this PR solve?
Rename page_num_list, top_list, position_list to page_num_int, top_int,
position_int
### Type of change
- [x] Refactoring
### What problem does this PR solve?
Handle infinity empty response. Close#3623
Show version in docker build log
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
Changed requirement to python 3.10.
Changed image base to Ubuntu 22.04 since it contains python 3.10.
### Type of change
- [x] Refactoring
### What problem does this PR solve?
Move the tiktoken of cl100k_base into docker image
issue: #3338
### Type of change
- [x] Refactoring
Signed-off-by: jinhai <haijin.chn@gmail.com>
Co-authored-by: Kevin Hu <kevinhu.sh@gmail.com>
### What problem does this PR solve?
This pull request implements the feature mentioned in #3322.
Instead of manually having to edit the `service_conf.yaml` file when
changes have been made to `.env` and mapping it into the docker
container at runtime, a template file is used and the values replaced by
the environment variables from the `.env` file when the container is
started.
### Type of change
- [X] New Feature (non-breaking change which adds functionality)
---------
Co-authored-by: Kevin Hu <kevinhu.sh@gmail.com>
Co-authored-by: Zhichang Yu <yuzhichang@gmail.com>
### What problem does this PR solve?
### Type of change
- [ ] Bug Fix (non-breaking change which fixes an issue)
- [x] New Feature (non-breaking change which adds functionality)
### What problem does this PR solve?
Added tika jar into image to avoid downloading. Close#3017
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
Added jdk to happify tika(https://pypi.org/project/tika/). The image
size become ~400MB bigger. Close#2886
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
### What problem does this PR solve?
feat: Delete http_api_reference.md from api folder #1102
### Type of change
- [x] New Feature (non-breaking change which adds functionality)
### What problem does this PR solve?
Fixed docker build
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [ ] New Feature (non-breaking change which adds functionality)
- [ ] Documentation Update
- [ ] Refactoring
- [ ] Performance Improvement
- [ ] Other (please describe):
### What problem does this PR solve?
1. Fix typo
2. Update comments.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
Signed-off-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
Modified download_deps.py
### Type of change
- [ ] Bug Fix (non-breaking change which fixes an issue)
- [ ] New Feature (non-breaking change which adds functionality)
- [ ] Documentation Update
- [ ] Refactoring
- [ ] Performance Improvement
- [x] Other (please describe): CI
### What problem does this PR solve?
build multi-arch image
### Type of change
- [ ] Bug Fix (non-breaking change which fixes an issue)
- [ ] New Feature (non-breaking change which adds functionality)
- [ ] Documentation Update
- [ ] Refactoring
- [ ] Performance Improvement
- [x] Other (please describe): CI
### What problem does this PR solve?
Fix poetry cache
### Type of change
- [ ] Bug Fix (non-breaking change which fixes an issue)
- [ ] New Feature (non-breaking change which adds functionality)
- [ ] Documentation Update
- [ ] Refactoring
- [ ] Performance Improvement
- [x] Other (please describe): CI
### What problem does this PR solve?
Fix Dockerfile for arm64
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [ ] New Feature (non-breaking change which adds functionality)
- [ ] Documentation Update
- [ ] Refactoring
- [ ] Performance Improvement
- [ ] Other (please describe):
---------
Co-authored-by: Ubuntu <ubuntu@arm-test.us-central1-f.c.ragflow-01.internal>
### What problem does this PR solve?
Updated Dockefile to use cache
### Type of change
- [ ] Bug Fix (non-breaking change which fixes an issue)
- [ ] New Feature (non-breaking change which adds functionality)
- [ ] Documentation Update
- [ ] Refactoring
- [ ] Performance Improvement
- [x] Other (please describe): CI
### What problem does this PR solve?
Updated Dockerfile
### Type of change
- [ ] Bug Fix (non-breaking change which fixes an issue)
- [ ] New Feature (non-breaking change which adds functionality)
- [ ] Documentation Update
- [x] Refactoring
- [ ] Performance Improvement
- [ ] Other (please describe):
### What problem does this PR solve?
Fix docker file bugs
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
- [x] Documentation Update
---------
Co-authored-by: writinwaters <93570324+writinwaters@users.noreply.github.com>
Co-authored-by: Kevin Hu <kevinhu.sh@gmail.com>
Add doc for dev image
### Type of change
- [x] Documentation Update
---------
Co-authored-by: writinwaters <93570324+writinwaters@users.noreply.github.com>
