diff --git a/Dockerfile b/Dockerfile index 6fd599d6b..6e0a1a831 100644 --- a/Dockerfile +++ b/Dockerfile @@ -129,6 +129,7 @@ ENV PATH="${VIRTUAL_ENV}/bin:${PATH}" ENV PYTHONPATH=/ragflow/ +COPY docker/service_conf.yaml.template ./conf/service_conf.yaml.template COPY docker/entrypoint.sh ./entrypoint.sh RUN chmod +x ./entrypoint.sh diff --git a/Dockerfile.scratch.oc9 b/Dockerfile.scratch.oc9 index 885041171..50a93a6a0 100644 --- a/Dockerfile.scratch.oc9 +++ b/Dockerfile.scratch.oc9 @@ -53,6 +53,7 @@ RUN conda run -n py11 python -m nltk.downloader wordnet ENV PYTHONPATH=/ragflow/ ENV HF_ENDPOINT=https://hf-mirror.com +COPY docker/service_conf.yaml.template ./conf/service_conf.yaml.template ADD docker/entrypoint.sh ./entrypoint.sh RUN chmod +x ./entrypoint.sh diff --git a/Dockerfile.slim b/Dockerfile.slim index d30baed9d..6b093db87 100644 --- a/Dockerfile.slim +++ b/Dockerfile.slim @@ -122,6 +122,7 @@ ENV PATH="${VIRTUAL_ENV}/bin:${PATH}" ENV PYTHONPATH=/ragflow/ +COPY docker/service_conf.yaml.template ./conf/service_conf.yaml.template COPY docker/entrypoint.sh ./entrypoint.sh RUN chmod +x ./entrypoint.sh diff --git a/README.md b/README.md index bfae7f929..109a32d92 100644 --- a/README.md +++ b/README.md @@ -216,7 +216,7 @@ releases! 🌟 5. In your web browser, enter the IP address of your server and log in to RAGFlow. > With the default settings, you only need to enter `http://IP_OF_YOUR_MACHINE` (**sans** port number) as the default HTTP serving port `80` can be omitted when using the default configurations. -6. In [service_conf.yaml](./docker/service_conf.yaml), select the desired LLM factory in `user_default_llm` and update +6. In [service_conf.yaml.template](./docker/service_conf.yaml.template), select the desired LLM factory in `user_default_llm` and update the `API_KEY` field with the corresponding API key. > See [llm_api_key_setup](https://ragflow.io/docs/dev/llm_api_key_setup) for more information. @@ -229,16 +229,11 @@ When it comes to system configurations, you will need to manage the following fi - [.env](./docker/.env): Keeps the fundamental setups for the system, such as `SVR_HTTP_PORT`, `MYSQL_PASSWORD`, and `MINIO_PASSWORD`. -- [service_conf.yaml](./docker/service_conf.yaml): Configures the back-end services. -- [docker-compose.yml](./docker/docker-compose.yml): The system relies - on [docker-compose.yml](./docker/docker-compose.yml) to start up. - -You must ensure that changes to the [.env](./docker/.env) file are in line with what are in the [service_conf.yaml](./docker/service_conf.yaml) file. +- [service_conf.yaml.template](./docker/service_conf.yaml.template): Configures the back-end services. The environment variables in this file will be automatically populated when the Docker container starts. Any environment variables set within the Docker container will be available for use, allowing you to customize service behavior based on the deployment environment. +- [docker-compose.yml](./docker/docker-compose.yml): The system relies on [docker-compose.yml](./docker/docker-compose.yml) to start up. > The [./docker/README](./docker/README.md) file provides a detailed description of the environment settings and service -> configurations, and you are REQUIRED to ensure that all environment settings listed in -> the [./docker/README](./docker/README.md) file are aligned with the corresponding configurations in -> the [service_conf.yaml](./docker/service_conf.yaml) file. +> configurations which can be used as `${ENV_VARS}` in the [service_conf.yaml.template](./docker/service_conf.yaml.template) file. To update the default HTTP serving port (80), go to [docker-compose.yml](./docker/docker-compose.yml) and change `80:80` to `:80`. @@ -293,11 +288,11 @@ docker build -f Dockerfile -t infiniflow/ragflow:dev . docker compose -f docker/docker-compose-base.yml up -d ``` - Add the following line to `/etc/hosts` to resolve all hosts specified in **docker/service_conf.yaml** to `127.0.0.1`: + Add the following line to `/etc/hosts` to resolve all hosts specified in **docker/.env** to `127.0.0.1`: ``` 127.0.0.1 es01 infinity mysql minio redis ``` - In **docker/service_conf.yaml**, update mysql port to `5455` and es port to `1200`, as specified in **docker/.env**. + In **docker/service_conf.yaml.template**, update mysql port to `5455` and es port to `1200`, as specified in **docker/.env**. 4. If you cannot access HuggingFace, set the `HF_ENDPOINT` environment variable to use a mirror site: diff --git a/docker/.env b/docker/.env index b8d82a32b..8ad572f0c 100644 --- a/docker/.env +++ b/docker/.env @@ -1,6 +1,9 @@ # The version of Elasticsearch. STACK_VERSION=8.11.3 +# The hostname where the Elasticsearch service is exposed +ES_HOST=es01 + # The port used to expose the Elasticsearch service to the host machine, # allowing EXTERNAL access to the service running inside the Docker container. ES_PORT=1200 @@ -27,10 +30,16 @@ INFINITY_PSQL_PORT=5432 # The password for MySQL. # When updated, you must revise the `mysql.password` entry in service_conf.yaml. MYSQL_PASSWORD=infini_rag_flow +# The hostname where the MySQL service is exposed +MYSQL_HOST=mysql +# The database of the MySQL service to use +MYSQL_DBNAME=rag_flow # The port used to expose the MySQL service to the host machine, # allowing EXTERNAL access to the MySQL database running inside the Docker container. MYSQL_PORT=5455 +# The hostname where the MySQL service is exposed +MINIO_HOST=minio # The port used to expose the MinIO console interface to the host machine, # allowing EXTERNAL access to the web-based console running inside the Docker container. MINIO_CONSOLE_PORT=9001 @@ -44,6 +53,8 @@ MINIO_USER=rag_flow # When updated, you must revise the `minio.password` entry in service_conf.yaml accordingly. MINIO_PASSWORD=infini_rag_flow +# The hostname where the Redis service is exposed +REDIS_HOST=redis # The port used to expose the Redis service to the host machine, # allowing EXTERNAL access to the Redis service running inside the Docker container. REDIS_PORT=6379 diff --git a/docker/README.md b/docker/README.md index 6731d6586..cbb63e38d 100644 --- a/docker/README.md +++ b/docker/README.md @@ -27,7 +27,7 @@ The [.env](./.env) file contains important environment variables for Docker. - `ES_PORT` The port used to expose the Elasticsearch service to the host machine, allowing **external** access to the service running inside the Docker container. Defaults to `1200`. - `ELASTIC_PASSWORD` - The password for Elasticsearch. When updated, you must revise the `es.password` entry in [service_conf.yaml](./service_conf.yaml) accordingly. + The password for Elasticsearch. ### Kibana @@ -46,7 +46,7 @@ The [.env](./.env) file contains important environment variables for Docker. ### MySQL - `MYSQL_PASSWORD` - The password for MySQL. When updated, you must revise the `mysql.password` entry in [service_conf.yaml](./service_conf.yaml) accordingly. + The password for MySQL. - `MYSQL_PORT` The port used to expose the MySQL service to the host machine, allowing **external** access to the MySQL database running inside the Docker container. Defaults to `5455`. @@ -57,16 +57,16 @@ The [.env](./.env) file contains important environment variables for Docker. - `MINIO_PORT` The port used to expose the MinIO API service to the host machine, allowing **external** access to the MinIO object storage service running inside the Docker container. Defaults to `9000`. - `MINIO_USER` - The username for MinIO. When updated, you must revise the `minio.user` entry in [service_conf.yaml](./service_conf.yaml) accordingly. + The username for MinIO. - `MINIO_PASSWORD` - The password for MinIO. When updated, you must revise the `minio.password` entry in [service_conf.yaml](./service_conf.yaml) accordingly. + The password for MinIO. ### Redis - `REDIS_PORT` The port used to expose the Redis service to the host machine, allowing **external** access to the Redis service running inside the Docker container. Defaults to `6379`. - `REDIS_PASSWORD` - The password for Redis. When updated, you must revise the `redis.password` entry in [service_conf.yaml](./service_conf.yaml) accordingly. + The password for Redis. ### RAGFlow @@ -119,7 +119,7 @@ The [.env](./.env) file contains important environment variables for Docker. ## 🐋 Service configuration -[service_conf.yaml](./service_conf.yaml) specifies the system-level configuration for RAGFlow and is used by its API server and task executor. +[service_conf.yaml](./service_conf.yaml) specifies the system-level configuration for RAGFlow and is used by its API server and task executor. In a dockerized setup, this file is automatically created based on the [service_conf.yaml.template](./service_conf.yaml.template) file (replacing all environment variables by their values). - `ragflow` - `host`: The API server's IP address inside the Docker container. Defaults to `0.0.0.0`. @@ -139,11 +139,11 @@ The [.env](./.env) file contains important environment variables for Docker. - `host`: The MinIO serving IP *and* port inside the Docker container. Defaults to `minio:9000`. - `oauth` - The OAuth configuration for signing up or signing in to RAGFlow using a third-party account. It is disabled by default. To enable this feature, uncomment the corresponding lines in **service_conf.yaml**. + The OAuth configuration for signing up or signing in to RAGFlow using a third-party account. It is disabled by default. To enable this feature, uncomment the corresponding lines in **service_conf.yaml.template**. - `github`: The GitHub authentication settings for your application. Visit the [Github Developer Settings page](https://github.com/settings/developers) to obtain your client_id and secret_key. - `user_default_llm` - The default LLM to use for a new RAGFlow user. It is disabled by default. To enable this feature, uncomment the corresponding lines in **service_conf.yaml**. + The default LLM to use for a new RAGFlow user. It is disabled by default. To enable this feature, uncomment the corresponding lines in **service_conf.yaml.template**. - `factory`: The LLM supplier. Available options: - `"OpenAI"` - `"DeepSeek"` diff --git a/docker/docker-compose-gpu.yml b/docker/docker-compose-gpu.yml index fd26869a8..1f0fe8a4c 100644 --- a/docker/docker-compose-gpu.yml +++ b/docker/docker-compose-gpu.yml @@ -16,7 +16,6 @@ services: - 80:80 - 443:443 volumes: - - ./service_conf.yaml:/ragflow/conf/service_conf.yaml - ./ragflow-logs:/ragflow/logs - ./nginx/ragflow.conf:/etc/nginx/conf.d/ragflow.conf - ./nginx/proxy.conf:/etc/nginx/proxy.conf diff --git a/docker/entrypoint.sh b/docker/entrypoint.sh index 1c2c3bc35..792bbca7a 100755 --- a/docker/entrypoint.sh +++ b/docker/entrypoint.sh @@ -1,5 +1,12 @@ #!/bin/bash +# replace env variables in the service_conf.yaml file +rm -rf /ragflow/conf/service_conf.yaml +while IFS= read -r line || [[ -n "$line" ]]; do + # Use eval to interpret the variable with default values + eval "echo \"$line\"" >> /ragflow/conf/service_conf.yaml +done < /ragflow/conf/service_conf.yaml.template + # unset http proxy which maybe set by docker daemon export http_proxy=""; export https_proxy=""; export no_proxy=""; export HTTP_PROXY=""; export HTTPS_PROXY=""; export NO_PROXY="" diff --git a/docker/service_conf.yaml.template b/docker/service_conf.yaml.template new file mode 100644 index 000000000..c0d3a03eb --- /dev/null +++ b/docker/service_conf.yaml.template @@ -0,0 +1,74 @@ +ragflow: + host: ${RAGFLOW_HOST:-0.0.0.0} + http_port: 9380 +mysql: + name: '${MYSQL_DBNAME:-rag_flow}' + user: '${MYSQL_USER:-root}' + password: '${MYSQL_PASSWORD:-infini_rag_flow}' + host: '${MYSQL_HOST:-mysql}' + port: 3306 + max_connections: 100 + stale_timeout: 30 +minio: + user: '${MINIO_USER:-rag_flow}' + password: '${MINIO_PASSWORD:-infini_rag_flow}' + host: '${MINIO_HOST:-minio}:9000' +es: + hosts: 'http://${ES_HOST:-es01}:9200' + username: '${ES_USER:-elastic}' + password: '${ES_PASSWORD:-infini_rag_flow}' +redis: + db: 1 + password: '${REDIS_PASSWORD:-infini_rag_flow}' + host: '${REDIS_HOST:-redis}:6379' + +# postgres: +# name: '${POSTGRES_DBNAME:-rag_flow}' +# user: '${POSTGRES_USER:-rag_flow}' +# password: '${POSTGRES_PASSWORD:-infini_rag_flow}' +# host: '${POSTGRES_HOST:-postgres}' +# port: 5432 +# max_connections: 100 +# stale_timeout: 30 +# s3: +# endpoint: 'endpoint' +# access_key: 'access_key' +# secret_key: 'secret_key' +# region: 'region' +# azure: +# auth_type: 'sas' +# container_url: 'container_url' +# sas_token: 'sas_token' +# azure: +# auth_type: 'spn' +# account_url: 'account_url' +# client_id: 'client_id' +# secret: 'secret' +# tenant_id: 'tenant_id' +# container_name: 'container_name' +# user_default_llm: +# factory: 'Tongyi-Qianwen' +# api_key: 'sk-xxxxxxxxxxxxx' +# base_url: '' +# oauth: +# github: +# client_id: xxxxxxxxxxxxxxxxxxxxxxxxx +# secret_key: xxxxxxxxxxxxxxxxxxxxxxxxxxxx +# url: https://github.com/login/oauth/access_token +# feishu: +# app_id: cli_xxxxxxxxxxxxxxxxxxx +# app_secret: xxxxxxxxxxxxxxxxxxxxxxxxxxxx +# app_access_token_url: https://open.feishu.cn/open-apis/auth/v3/app_access_token/internal +# user_access_token_url: https://open.feishu.cn/open-apis/authen/v1/oidc/access_token +# grant_type: 'authorization_code' +# authentication: +# client: +# switch: false +# http_app_key: +# http_secret_key: +# site: +# switch: false +# permission: +# switch: false +# component: false +# dataset: false diff --git a/docs/configurations.md b/docs/configurations.md index 315f608a6..6dc7619fa 100644 --- a/docs/configurations.md +++ b/docs/configurations.md @@ -12,11 +12,9 @@ Configurations for installing RAGFlow via Docker. When it comes to system configurations, you will need to manage the following files: - [.env](https://github.com/infiniflow/ragflow/blob/main/docker/.env): Contains important environment variables for Docker. -- [service_conf.yaml](https://github.com/infiniflow/ragflow/blob/main/docker/service_conf.yaml): Configures the back-end services. It specifies the system-level configuration for RAGFlow and is used by its API server and task executor. +- [service_conf.yaml.template](https://github.com/infiniflow/ragflow/blob/main/docker/service_conf.yaml.template): Configures the back-end services. It specifies the system-level configuration for RAGFlow and is used by its API server and task executor. Upon container startup, the `service_conf.yaml` file will be generated based on this template file. This process replaces any environment variables within the template, allowing for dynamic configuration tailored to the container's environment. - [docker-compose.yml](https://github.com/infiniflow/ragflow/blob/main/docker/docker-compose.yml): The Docker Compose file for starting up the RAGFlow service. -You must ensure that changes to the [.env](https://github.com/infiniflow/ragflow/blob/main/docker/.env) file are in line with what are in the [service_conf.yaml](https://github.com/infiniflow/ragflow/blob/main/docker/service_conf.yaml) file. - To update the default HTTP serving port (80), go to [docker-compose.yml](./docker/docker-compose.yml) and change `80:80` to `:80`. @@ -47,7 +45,7 @@ The [.env](https://github.com/infiniflow/ragflow/blob/main/docker/.env) file con - `ES_PORT` The port used to expose the Elasticsearch service to the host machine, allowing **external** access to the service running inside the Docker container. Defaults to `1200`. - `ELASTIC_PASSWORD` - The password for Elasticsearch. When updated, you must revise the `es.password` entry in [service_conf.yaml](https://github.com/infiniflow/ragflow/blob/main/docker/service_conf.yaml) accordingly. + The password for Elasticsearch. ### Kibana @@ -66,7 +64,7 @@ The [.env](https://github.com/infiniflow/ragflow/blob/main/docker/.env) file con ### MySQL - `MYSQL_PASSWORD` - The password for MySQL. When updated, you must revise the `mysql.password` entry in [service_conf.yaml](https://github.com/infiniflow/ragflow/blob/main/docker/service_conf.yaml) accordingly. + The password for MySQL. - `MYSQL_PORT` The port used to expose the MySQL service to the host machine, allowing **external** access to the MySQL database running inside the Docker container. Defaults to `5455`. @@ -77,16 +75,16 @@ The [.env](https://github.com/infiniflow/ragflow/blob/main/docker/.env) file con - `MINIO_PORT` The port used to expose the MinIO API service to the host machine, allowing **external** access to the MinIO object storage service running inside the Docker container. Defaults to `9000`. - `MINIO_USER` - The username for MinIO. When updated, you must revise the `minio.user` entry in [service_conf.yaml](https://github.com/infiniflow/ragflow/blob/main/docker/service_conf.yaml) accordingly. + The username for MinIO. - `MINIO_PASSWORD` - The password for MinIO. When updated, you must revise the `minio.password` entry in [service_conf.yaml](https://github.com/infiniflow/ragflow/blob/main/docker/service_conf.yaml) accordingly. + The password for MinIO. accordingly. ### Redis - `REDIS_PORT` The port used to expose the Redis service to the host machine, allowing **external** access to the Redis service running inside the Docker container. Defaults to `6379`. - `REDIS_PASSWORD` - The password for Redis. When updated, you must revise the `redis.password` entry in [service_conf.yaml](https://github.com/infiniflow/ragflow/blob/main/docker/service_conf.yaml) accordingly. + The password for Redis. ### RAGFlow @@ -140,7 +138,7 @@ If you cannot download the RAGFlow Docker image, try the following mirrors. ## Service configuration -[service_conf.yaml](https://github.com/infiniflow/ragflow/blob/main/docker/service_conf.yaml) specifies the system-level configuration for RAGFlow and is used by its API server and task executor. +[service_conf.yaml.template](https://github.com/infiniflow/ragflow/blob/main/docker/service_conf.yaml.template) specifies the system-level configuration for RAGFlow and is used by its API server and task executor. ### `ragflow` @@ -164,13 +162,13 @@ If you cannot download the RAGFlow Docker image, try the following mirrors. ### `oauth` -The OAuth configuration for signing up or signing in to RAGFlow using a third-party account. It is disabled by default. To enable this feature, uncomment the corresponding lines in **service_conf.yaml**. +The OAuth configuration for signing up or signing in to RAGFlow using a third-party account. It is disabled by default. To enable this feature, uncomment the corresponding lines in **service_conf.yaml.template**. - `github`: The GitHub authentication settings for your application. Visit the [Github Developer Settings](https://github.com/settings/developers) page to obtain your client_id and secret_key. ### `user_default_llm` -The default LLM to use for a new RAGFlow user. It is disabled by default. To enable this feature, uncomment the corresponding lines in **service_conf.yaml**. +The default LLM to use for a new RAGFlow user. It is disabled by default. To enable this feature, uncomment the corresponding lines in **service_conf.yaml.template**. - `factory`: The LLM supplier. Available options: - `"OpenAI"` diff --git a/docs/guides/develop/launch_ragflow_from_source.md b/docs/guides/develop/launch_ragflow_from_source.md index 160c704ab..05af7c668 100644 --- a/docs/guides/develop/launch_ragflow_from_source.md +++ b/docs/guides/develop/launch_ragflow_from_source.md @@ -64,13 +64,13 @@ docker compose -f docker/docker-compose-base.yml up -d ### Update `host` and `port` Settings for Third-party Services -1. Add the following line to `/etc/hosts` to resolve all hosts specified in **docker/service_conf.yaml** to `127.0.0.1`: +1. Add the following line to `/etc/hosts` to resolve all hosts specified in **docker/service_conf.yaml.template** to `127.0.0.1`: ``` 127.0.0.1 es01 infinity mysql minio redis ``` -2. In **docker/service_conf.yaml**, update mysql port to `5455` and es port to `1200`, as specified in **docker/.env**. +2. In **docker/service_conf.yaml.template**, update mysql port to `5455` and es port to `1200`, as specified in **docker/.env**. ### Launch the RAGFlow Backend Service diff --git a/docs/guides/llm_api_key_setup.md b/docs/guides/llm_api_key_setup.md index 2d9c9563a..89e2dd73e 100644 --- a/docs/guides/llm_api_key_setup.md +++ b/docs/guides/llm_api_key_setup.md @@ -19,7 +19,7 @@ If you find your online LLM is not on the list, don't feel disheartened. The lis You have two options for configuring your model API key: -- Configure it in **service_conf.yaml** before starting RAGFlow. +- Configure it in **service_conf.yaml.template** before starting RAGFlow. - Configure it on the **Model Providers** page after logging into RAGFlow. ### Configure model API key before starting up RAGFlow @@ -37,7 +37,7 @@ You have two options for configuring your model API key: ### Configure model API key after logging into RAGFlow :::caution WARNING -After logging into RAGFlow, configuring your model API key through the **service_conf.yaml** file will no longer take effect. +After logging into RAGFlow, configuring your model API key through the **service_conf.yaml.template** file will no longer take effect. ::: After logging into RAGFlow, you can *only* configure API Key on the **Model Providers** page: