mirror of
https://github.com/rasbt/LLMs-from-scratch.git
synced 2025-08-28 18:40:01 +00:00
Simplify the setup in CI (#542)
* Simplify the setup in CI * windows --no-binary option * windows uv workflow * add path * some fixes * some fixes * some fixes * cleanup * cleanup * cleanup * cleanup * cleanup * more windows issues * more windows issues * more windows issues * more windows issues * more windows issues * more windows issues * more windows issues * more windows issues * revert
This commit is contained in:
parent
4fad4695f6
commit
11801be0e9
5
.github/workflows/basic-tests-linux-uv.yml
vendored
5
.github/workflows/basic-tests-linux-uv.yml
vendored
@ -33,14 +33,13 @@ jobs:
|
||||
- name: Set up Python (uv)
|
||||
uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: "3.11"
|
||||
python-version: "3.13"
|
||||
|
||||
- name: Install uv and dependencies
|
||||
shell: bash
|
||||
run: |
|
||||
curl -LsSf https://astral.sh/uv/install.sh | sh
|
||||
uv python install 3.11
|
||||
uv add . --dev
|
||||
uv sync --dev --python=3.10 # tests for backwards compatibility
|
||||
uv pip install -r ch05/07_gpt_to_llama/tests/test-requirements-extra.txt
|
||||
uv add pytest-ruff nbval
|
||||
|
||||
|
5
.github/workflows/basic-tests-macos-uv.yml
vendored
5
.github/workflows/basic-tests-macos-uv.yml
vendored
@ -33,14 +33,13 @@ jobs:
|
||||
- name: Set up Python (uv)
|
||||
uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: "3.11"
|
||||
python-version: "3.13"
|
||||
|
||||
- name: Install uv and dependencies
|
||||
shell: bash
|
||||
run: |
|
||||
curl -LsSf https://astral.sh/uv/install.sh | sh
|
||||
uv python install 3.11
|
||||
uv add . --dev
|
||||
uv sync --dev --python=3.10 # tests for backwards compatibility
|
||||
uv pip install -r ch05/07_gpt_to_llama/tests/test-requirements-extra.txt
|
||||
uv add pytest-ruff nbval
|
||||
|
||||
|
@ -31,13 +31,12 @@ jobs:
|
||||
- name: Set up Python
|
||||
uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: "3.11"
|
||||
python-version: "3.13"
|
||||
|
||||
- name: Install dependencies
|
||||
run: |
|
||||
curl -LsSf https://astral.sh/uv/install.sh | sh
|
||||
uv python install 3.11
|
||||
uv add . --dev
|
||||
uv sync --dev --python=3.10 # tests for backwards compatibility
|
||||
uv pip install -r ch05/07_gpt_to_llama/tests/test-requirements-extra.txt
|
||||
uv add torch==${{ matrix.pytorch-version }}
|
||||
uv add pytest-ruff nbval
|
||||
|
2
.github/workflows/basic-tests-pip.yml
vendored
2
.github/workflows/basic-tests-pip.yml
vendored
@ -33,7 +33,7 @@ jobs:
|
||||
- name: Set up Python
|
||||
uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: "3.11"
|
||||
python-version: "3.10" # tests for backwards compatibility
|
||||
|
||||
- name: Create Virtual Environment and Install Dependencies
|
||||
run: |
|
||||
|
5
.github/workflows/basic-tests-pytorch-rc.yml
vendored
5
.github/workflows/basic-tests-pytorch-rc.yml
vendored
@ -27,13 +27,12 @@ jobs:
|
||||
- name: Set up Python
|
||||
uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: "3.11"
|
||||
python-version: "3.13"
|
||||
|
||||
- name: Install dependencies
|
||||
run: |
|
||||
curl -LsSf https://astral.sh/uv/install.sh | sh
|
||||
uv python install 3.11
|
||||
uv add . --dev
|
||||
uv sync --dev --python=3.10 # tests for backwards compatibility
|
||||
uv pip install -r ch05/07_gpt_to_llama/tests/test-requirements-extra.txt
|
||||
uv add pytest-ruff nbval
|
||||
uv pip install --pre torch torchvision torchaudio --index-url https://download.pytorch.org/whl/nightly/cpu
|
||||
|
66
.github/workflows/basic-tests-windows-uv-pip.yml.disabled
vendored
Normal file
66
.github/workflows/basic-tests-windows-uv-pip.yml.disabled
vendored
Normal file
@ -0,0 +1,66 @@
|
||||
name: Code tests Windows (uv/pip)
|
||||
|
||||
on:
|
||||
push:
|
||||
branches: [ main ]
|
||||
paths:
|
||||
- '**/*.py'
|
||||
- '**/*.ipynb'
|
||||
- '**/*.yaml'
|
||||
- '**/*.yml'
|
||||
- '**/*.sh'
|
||||
pull_request:
|
||||
branches: [ main ]
|
||||
paths:
|
||||
- '**/*.py'
|
||||
- '**/*.ipynb'
|
||||
- '**/*.yaml'
|
||||
- '**/*.yml'
|
||||
- '**/*.sh'
|
||||
|
||||
jobs:
|
||||
test:
|
||||
runs-on: windows-latest
|
||||
|
||||
steps:
|
||||
- name: Checkout Code
|
||||
uses: actions/checkout@v4
|
||||
|
||||
- name: Set up Python
|
||||
uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: "3.13"
|
||||
|
||||
- name: Install dependencies
|
||||
shell: pwsh
|
||||
run: |
|
||||
$env:Path = "C:\Users\runneradmin\.local\bin;$env:Path"
|
||||
python -m pip install --upgrade pip
|
||||
python -m pip install uv
|
||||
uv venv --python=python3.11
|
||||
. .\.venv\Scripts\Activate.ps1
|
||||
$env:UV_PIP_OPTS="--no-binary tensorflow-io-gcs-filesystem"
|
||||
uv pip install -r requirements.txt
|
||||
uv pip install -r ch05/07_gpt_to_llama/tests/test-requirements-extra.txt
|
||||
uv pip install pytest-ruff nbval
|
||||
uv pip install --force-reinstall matplotlib "numpy<2.1"
|
||||
|
||||
- name: Run Python Tests
|
||||
shell: pwsh
|
||||
run: |
|
||||
$env:Path = "C:\Users\runneradmin\.local\bin;$env:Path"
|
||||
. .\.venv\Scripts\Activate.ps1
|
||||
pytest --ruff setup/02_installing-python-libraries/tests.py
|
||||
pytest --ruff ch04/01_main-chapter-code/tests.py
|
||||
pytest --ruff ch05/01_main-chapter-code/tests.py
|
||||
pytest --ruff ch05/07_gpt_to_llama/tests/tests.py
|
||||
pytest --ruff ch06/01_main-chapter-code/tests.py
|
||||
|
||||
- name: Run Jupyter Notebook Tests
|
||||
shell: pwsh
|
||||
run: |
|
||||
$env:Path = "C:\Users\runneradmin\.local\bin;$env:Path"
|
||||
. .\.venv\Scripts\Activate.ps1
|
||||
pytest --ruff --nbval ch02/01_main-chapter-code/dataloader.ipynb
|
||||
pytest --ruff --nbval ch03/01_main-chapter-code/multihead-attention.ipynb
|
||||
pytest --ruff --nbval ch02/04_bonus_dataloader-intuition/dataloader-intuition.ipynb
|
63
.github/workflows/basic-tests-windows-uv.yml.disabled
vendored
Normal file
63
.github/workflows/basic-tests-windows-uv.yml.disabled
vendored
Normal file
@ -0,0 +1,63 @@
|
||||
name: Code tests Windows (uv)
|
||||
|
||||
on:
|
||||
push:
|
||||
branches: [ main ]
|
||||
paths:
|
||||
- '**/*.py'
|
||||
- '**/*.ipynb'
|
||||
- '**/*.yaml'
|
||||
- '**/*.yml'
|
||||
- '**/*.sh'
|
||||
pull_request:
|
||||
branches: [ main ]
|
||||
paths:
|
||||
- '**/*.py'
|
||||
- '**/*.ipynb'
|
||||
- '**/*.yaml'
|
||||
- '**/*.yml'
|
||||
- '**/*.sh'
|
||||
|
||||
jobs:
|
||||
test:
|
||||
runs-on: windows-latest
|
||||
|
||||
steps:
|
||||
- name: Checkout Code
|
||||
uses: actions/checkout@v4
|
||||
|
||||
- name: Set up Python
|
||||
uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: "3.13"
|
||||
|
||||
- name: Install dependencies
|
||||
shell: pwsh
|
||||
run: |
|
||||
# Prepend local bin directory to PATH
|
||||
powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex"
|
||||
$env:Path = "C:\Users\runneradmin\.local\bin;$env:Path"
|
||||
uv sync --dev --python=3.10
|
||||
$env:UV_PIP_OPTS="--no-binary tensorflow-io-gcs-filesystem"
|
||||
uv pip install -r requirements.txt
|
||||
uv pip install matplotlib # for some reason Windows requires this
|
||||
uv pip install -r ch05/07_gpt_to_llama/tests/test-requirements-extra.txt
|
||||
uv add pytest-ruff nbval
|
||||
|
||||
- name: Run Python Tests
|
||||
shell: pwsh
|
||||
run: |
|
||||
. .\.venv\Scripts\Activate.ps1
|
||||
pytest --ruff setup/02_installing-python-libraries/tests.py
|
||||
pytest --ruff ch04/01_main-chapter-code/tests.py
|
||||
pytest --ruff ch05/01_main-chapter-code/tests.py
|
||||
pytest --ruff ch05/07_gpt_to_llama/tests/tests.py
|
||||
pytest --ruff ch06/01_main-chapter-code/tests.py
|
||||
|
||||
- name: Run Jupyter Notebook Tests
|
||||
shell: pwsh
|
||||
run: |
|
||||
. .\.venv\Scripts\Activate.ps1
|
||||
pytest --ruff --nbval ch02/01_main-chapter-code/dataloader.ipynb
|
||||
pytest --ruff --nbval ch03/01_main-chapter-code/multihead-attention.ipynb
|
||||
pytest --ruff --nbval ch02/04_bonus_dataloader-intuition/dataloader-intuition.ipynb
|
4
.github/workflows/check-links.yml
vendored
4
.github/workflows/check-links.yml
vendored
@ -18,13 +18,11 @@ jobs:
|
||||
- name: Set up Python
|
||||
uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: '3.11'
|
||||
python-version: "3.10"
|
||||
|
||||
- name: Install dependencies
|
||||
run: |
|
||||
curl -LsSf https://astral.sh/uv/install.sh | sh
|
||||
uv python install 3.11
|
||||
uv add . --dev
|
||||
uv add pytest-ruff pytest-check-links
|
||||
# Current version of retry doesn't work well if there are broken non-URL links
|
||||
# pip install pytest pytest-check-links pytest-retry
|
||||
|
5
.github/workflows/check-spelling-errors.yml
vendored
5
.github/workflows/check-spelling-errors.yml
vendored
@ -18,13 +18,12 @@ jobs:
|
||||
- name: Set up Python
|
||||
uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: '3.11'
|
||||
python-version: "3.10"
|
||||
|
||||
- name: Install codespell
|
||||
run: |
|
||||
curl -LsSf https://astral.sh/uv/install.sh | sh
|
||||
uv python install 3.11
|
||||
uv add . --dev
|
||||
uv sync --dev --python=3.10
|
||||
uv add codespell
|
||||
|
||||
- name: Run codespell
|
||||
|
5
.github/workflows/pep8-linter.yml
vendored
5
.github/workflows/pep8-linter.yml
vendored
@ -14,12 +14,11 @@ jobs:
|
||||
- name: Set up Python
|
||||
uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: '3.11'
|
||||
python-version: "3.13"
|
||||
- name: Install ruff (a faster flake 8 equivalent)
|
||||
run: |
|
||||
curl -LsSf https://astral.sh/uv/install.sh | sh
|
||||
uv python install 3.11
|
||||
uv add . --dev
|
||||
uv sync --dev --python=3.10
|
||||
uv add ruff
|
||||
|
||||
- name: Run ruff with exceptions
|
||||
|
@ -9,8 +9,7 @@ dependencies = [
|
||||
"jupyterlab>=4.0",
|
||||
"tiktoken>=0.5.1",
|
||||
"matplotlib>=3.7.1",
|
||||
"tensorflow>=2.18.0; sys_platform != \"win32\"",
|
||||
"tensorflow-cpu>=2.18.0; sys_platform == \"win32\"",
|
||||
"tensorflow>=2.18.0",
|
||||
"tqdm>=4.66.1",
|
||||
"numpy>=1.26,<2.1",
|
||||
"pandas>=2.2.1",
|
||||
|
@ -1,9 +1,8 @@
|
||||
torch >= 2.3.0 # all
|
||||
jupyterlab >= 4.0 # all
|
||||
tiktoken >= 0.5.1 # ch02; ch04; ch05
|
||||
matplotlib >= 3.7.1 # ch04; ch05
|
||||
tensorflow>=2.18.0; sys_platform != "win32" # ch05 (non-Windows)
|
||||
tensorflow-cpu>=2.18.0; sys_platform == "win32" # ch05 (Windows)
|
||||
matplotlib >= 3.7.1 # ch04; ch06; ch07
|
||||
tensorflow>=2.18.0 # ch05; ch06; ch07
|
||||
tqdm >= 4.66.1 # ch05; ch07
|
||||
numpy >= 1.26, < 2.1 # dependency of several other libraries like torch and pandas
|
||||
pandas >= 2.2.1 # ch06
|
||||
|
@ -50,7 +50,7 @@ If it returns 3.10 or newer, no further action is required.
|
||||
|
||||
|
||||
> [!NOTE]
|
||||
> I recommend installing a Python version that is at least 1-3 versions older than the most recent release to ensure PyTorch compatibility. For example, if the most recent version is Python 3.13, I recommend installing version 3.10, 3.11, or 3.12.
|
||||
> I recommend installing a Python version that is at least 2 versions older than the most recent release to ensure PyTorch compatibility. For example, if the most recent version is Python 3.13, I recommend installing version 3.10 or 3.11.
|
||||
|
||||
Otherwise, if Python is not installed or is an older version, you can install it for your operating system as described below.
|
||||
|
||||
@ -62,7 +62,7 @@ Otherwise, if Python is not installed or is an older version, you can install it
|
||||
|
||||
```bash
|
||||
sudo apt update
|
||||
sudo apt install python3.11 python3.11-venv python3.11-dev
|
||||
sudo apt install python3.10 python3.10-venv python3.10-dev
|
||||
```
|
||||
|
||||
<br>
|
||||
@ -143,13 +143,13 @@ uv pip install packaging
|
||||
To install all required packages from a `requirements.txt` file (such as the one located at the top level of this GitHub repository) run the following command, assuming the file is in the same directory as your terminal session:
|
||||
|
||||
```bash
|
||||
uv pip install -U -r requirements.txt
|
||||
uv pip install -r requirements.txt
|
||||
```
|
||||
|
||||
Alternatively, install the latest dependencies directly from the repository:
|
||||
|
||||
```bash
|
||||
uv pip install -U -r https://raw.githubusercontent.com/rasbt/LLMs-from-scratch/refs/heads/main/requirements.txt
|
||||
uv pip install -r https://raw.githubusercontent.com/rasbt/LLMs-from-scratch/refs/heads/main/requirements.txt
|
||||
```
|
||||
|
||||
|
||||
@ -246,7 +246,7 @@ conda create -n LLMs python=3.10
|
||||
|
||||
<img src="https://sebastianraschka.com/images/LLMs-from-scratch-images/setup/01_optional-python-setup-preferences/new-env.png" alt="new-env" width="600px">
|
||||
|
||||
> Many scientific computing libraries do not immediately support the newest version of Python. Therefore, when installing PyTorch, it's advisable to use a version of Python that is one to three releases older. For instance, if the latest version of Python is 3.13, using Python 3.10, 3.11, or 3.12 is recommended.
|
||||
> Many scientific computing libraries do not immediately support the newest version of Python. Therefore, when installing PyTorch, it's advisable to use a version of Python that is one or two releases older. For instance, if the latest version of Python is 3.13, using Python 3.10 or 3.11 is recommended.
|
||||
|
||||
Next, activate your new virtual environment (you have to do it every time you open a new terminal window or tab):
|
||||
|
||||
|
@ -70,6 +70,8 @@ uv sync --dev --python 3.11
|
||||
> If you have problems with the following commands above due to certain dependencies (for example, if you are using Windows), you can always fall back to regular pip:
|
||||
> `uv add pip`
|
||||
> `uv run python -m pip install -U -r requirements.txt`
|
||||
>
|
||||
> Since the TensorFo
|
||||
|
||||
|
||||
|
||||
|
Loading…
x
Reference in New Issue
Block a user