Simplify the setup in CI (#542)

* Simplify the setup in CI

* windows --no-binary option

* windows uv workflow

* add path

* some fixes

* some fixes

* some fixes

* cleanup

* cleanup

* cleanup

* cleanup

* cleanup

* more windows issues

* more windows issues

* more windows issues

* more windows issues

* more windows issues

* more windows issues

* more windows issues

* more windows issues

* revert
This commit is contained in:
Sebastian Raschka 2025-02-20 11:56:52 -06:00 committed by GitHub
parent 4fad4695f6
commit 11801be0e9
14 changed files with 160 additions and 39 deletions

View File

@ -33,14 +33,13 @@ jobs:
- name: Set up Python (uv)
uses: actions/setup-python@v5
with:
python-version: "3.11"
python-version: "3.13"
- name: Install uv and dependencies
shell: bash
run: |
curl -LsSf https://astral.sh/uv/install.sh | sh
uv python install 3.11
uv add . --dev
uv sync --dev --python=3.10 # tests for backwards compatibility
uv pip install -r ch05/07_gpt_to_llama/tests/test-requirements-extra.txt
uv add pytest-ruff nbval

View File

@ -33,14 +33,13 @@ jobs:
- name: Set up Python (uv)
uses: actions/setup-python@v5
with:
python-version: "3.11"
python-version: "3.13"
- name: Install uv and dependencies
shell: bash
run: |
curl -LsSf https://astral.sh/uv/install.sh | sh
uv python install 3.11
uv add . --dev
uv sync --dev --python=3.10 # tests for backwards compatibility
uv pip install -r ch05/07_gpt_to_llama/tests/test-requirements-extra.txt
uv add pytest-ruff nbval

View File

@ -31,13 +31,12 @@ jobs:
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: "3.11"
python-version: "3.13"
- name: Install dependencies
run: |
curl -LsSf https://astral.sh/uv/install.sh | sh
uv python install 3.11
uv add . --dev
uv sync --dev --python=3.10 # tests for backwards compatibility
uv pip install -r ch05/07_gpt_to_llama/tests/test-requirements-extra.txt
uv add torch==${{ matrix.pytorch-version }}
uv add pytest-ruff nbval

View File

@ -33,7 +33,7 @@ jobs:
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: "3.11"
python-version: "3.10" # tests for backwards compatibility
- name: Create Virtual Environment and Install Dependencies
run: |

View File

@ -27,13 +27,12 @@ jobs:
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: "3.11"
python-version: "3.13"
- name: Install dependencies
run: |
curl -LsSf https://astral.sh/uv/install.sh | sh
uv python install 3.11
uv add . --dev
uv sync --dev --python=3.10 # tests for backwards compatibility
uv pip install -r ch05/07_gpt_to_llama/tests/test-requirements-extra.txt
uv add pytest-ruff nbval
uv pip install --pre torch torchvision torchaudio --index-url https://download.pytorch.org/whl/nightly/cpu

View File

@ -0,0 +1,66 @@
name: Code tests Windows (uv/pip)
on:
push:
branches: [ main ]
paths:
- '**/*.py'
- '**/*.ipynb'
- '**/*.yaml'
- '**/*.yml'
- '**/*.sh'
pull_request:
branches: [ main ]
paths:
- '**/*.py'
- '**/*.ipynb'
- '**/*.yaml'
- '**/*.yml'
- '**/*.sh'
jobs:
test:
runs-on: windows-latest
steps:
- name: Checkout Code
uses: actions/checkout@v4
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: "3.13"
- name: Install dependencies
shell: pwsh
run: |
$env:Path = "C:\Users\runneradmin\.local\bin;$env:Path"
python -m pip install --upgrade pip
python -m pip install uv
uv venv --python=python3.11
. .\.venv\Scripts\Activate.ps1
$env:UV_PIP_OPTS="--no-binary tensorflow-io-gcs-filesystem"
uv pip install -r requirements.txt
uv pip install -r ch05/07_gpt_to_llama/tests/test-requirements-extra.txt
uv pip install pytest-ruff nbval
uv pip install --force-reinstall matplotlib "numpy<2.1"
- name: Run Python Tests
shell: pwsh
run: |
$env:Path = "C:\Users\runneradmin\.local\bin;$env:Path"
. .\.venv\Scripts\Activate.ps1
pytest --ruff setup/02_installing-python-libraries/tests.py
pytest --ruff ch04/01_main-chapter-code/tests.py
pytest --ruff ch05/01_main-chapter-code/tests.py
pytest --ruff ch05/07_gpt_to_llama/tests/tests.py
pytest --ruff ch06/01_main-chapter-code/tests.py
- name: Run Jupyter Notebook Tests
shell: pwsh
run: |
$env:Path = "C:\Users\runneradmin\.local\bin;$env:Path"
. .\.venv\Scripts\Activate.ps1
pytest --ruff --nbval ch02/01_main-chapter-code/dataloader.ipynb
pytest --ruff --nbval ch03/01_main-chapter-code/multihead-attention.ipynb
pytest --ruff --nbval ch02/04_bonus_dataloader-intuition/dataloader-intuition.ipynb

View File

@ -0,0 +1,63 @@
name: Code tests Windows (uv)
on:
push:
branches: [ main ]
paths:
- '**/*.py'
- '**/*.ipynb'
- '**/*.yaml'
- '**/*.yml'
- '**/*.sh'
pull_request:
branches: [ main ]
paths:
- '**/*.py'
- '**/*.ipynb'
- '**/*.yaml'
- '**/*.yml'
- '**/*.sh'
jobs:
test:
runs-on: windows-latest
steps:
- name: Checkout Code
uses: actions/checkout@v4
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: "3.13"
- name: Install dependencies
shell: pwsh
run: |
# Prepend local bin directory to PATH
powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex"
$env:Path = "C:\Users\runneradmin\.local\bin;$env:Path"
uv sync --dev --python=3.10
$env:UV_PIP_OPTS="--no-binary tensorflow-io-gcs-filesystem"
uv pip install -r requirements.txt
uv pip install matplotlib # for some reason Windows requires this
uv pip install -r ch05/07_gpt_to_llama/tests/test-requirements-extra.txt
uv add pytest-ruff nbval
- name: Run Python Tests
shell: pwsh
run: |
. .\.venv\Scripts\Activate.ps1
pytest --ruff setup/02_installing-python-libraries/tests.py
pytest --ruff ch04/01_main-chapter-code/tests.py
pytest --ruff ch05/01_main-chapter-code/tests.py
pytest --ruff ch05/07_gpt_to_llama/tests/tests.py
pytest --ruff ch06/01_main-chapter-code/tests.py
- name: Run Jupyter Notebook Tests
shell: pwsh
run: |
. .\.venv\Scripts\Activate.ps1
pytest --ruff --nbval ch02/01_main-chapter-code/dataloader.ipynb
pytest --ruff --nbval ch03/01_main-chapter-code/multihead-attention.ipynb
pytest --ruff --nbval ch02/04_bonus_dataloader-intuition/dataloader-intuition.ipynb

View File

@ -18,13 +18,11 @@ jobs:
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: '3.11'
python-version: "3.10"
- name: Install dependencies
run: |
curl -LsSf https://astral.sh/uv/install.sh | sh
uv python install 3.11
uv add . --dev
uv add pytest-ruff pytest-check-links
# Current version of retry doesn't work well if there are broken non-URL links
# pip install pytest pytest-check-links pytest-retry

View File

@ -18,13 +18,12 @@ jobs:
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: '3.11'
python-version: "3.10"
- name: Install codespell
run: |
curl -LsSf https://astral.sh/uv/install.sh | sh
uv python install 3.11
uv add . --dev
uv sync --dev --python=3.10
uv add codespell
- name: Run codespell

View File

@ -14,12 +14,11 @@ jobs:
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: '3.11'
python-version: "3.13"
- name: Install ruff (a faster flake 8 equivalent)
run: |
curl -LsSf https://astral.sh/uv/install.sh | sh
uv python install 3.11
uv add . --dev
uv sync --dev --python=3.10
uv add ruff
- name: Run ruff with exceptions

View File

@ -9,8 +9,7 @@ dependencies = [
"jupyterlab>=4.0",
"tiktoken>=0.5.1",
"matplotlib>=3.7.1",
"tensorflow>=2.18.0; sys_platform != \"win32\"",
"tensorflow-cpu>=2.18.0; sys_platform == \"win32\"",
"tensorflow>=2.18.0",
"tqdm>=4.66.1",
"numpy>=1.26,<2.1",
"pandas>=2.2.1",

View File

@ -1,9 +1,8 @@
torch >= 2.3.0 # all
jupyterlab >= 4.0 # all
tiktoken >= 0.5.1 # ch02; ch04; ch05
matplotlib >= 3.7.1 # ch04; ch05
tensorflow>=2.18.0; sys_platform != "win32" # ch05 (non-Windows)
tensorflow-cpu>=2.18.0; sys_platform == "win32" # ch05 (Windows)
matplotlib >= 3.7.1 # ch04; ch06; ch07
tensorflow>=2.18.0 # ch05; ch06; ch07
tqdm >= 4.66.1 # ch05; ch07
numpy >= 1.26, < 2.1 # dependency of several other libraries like torch and pandas
pandas >= 2.2.1 # ch06

View File

@ -50,7 +50,7 @@ If it returns 3.10 or newer, no further action is required.
&nbsp;
> [!NOTE]
> I recommend installing a Python version that is at least 1-3 versions older than the most recent release to ensure PyTorch compatibility. For example, if the most recent version is Python 3.13, I recommend installing version 3.10, 3.11, or 3.12.
> I recommend installing a Python version that is at least 2 versions older than the most recent release to ensure PyTorch compatibility. For example, if the most recent version is Python 3.13, I recommend installing version 3.10 or 3.11.
Otherwise, if Python is not installed or is an older version, you can install it for your operating system as described below.
@ -62,7 +62,7 @@ Otherwise, if Python is not installed or is an older version, you can install it
```bash
sudo apt update
sudo apt install python3.11 python3.11-venv python3.11-dev
sudo apt install python3.10 python3.10-venv python3.10-dev
```
<br>
@ -143,13 +143,13 @@ uv pip install packaging
To install all required packages from a `requirements.txt` file (such as the one located at the top level of this GitHub repository) run the following command, assuming the file is in the same directory as your terminal session:
```bash
uv pip install -U -r requirements.txt
uv pip install -r requirements.txt
```
Alternatively, install the latest dependencies directly from the repository:
```bash
uv pip install -U -r https://raw.githubusercontent.com/rasbt/LLMs-from-scratch/refs/heads/main/requirements.txt
uv pip install -r https://raw.githubusercontent.com/rasbt/LLMs-from-scratch/refs/heads/main/requirements.txt
```
@ -246,7 +246,7 @@ conda create -n LLMs python=3.10
<img src="https://sebastianraschka.com/images/LLMs-from-scratch-images/setup/01_optional-python-setup-preferences/new-env.png" alt="new-env" width="600px">
> Many scientific computing libraries do not immediately support the newest version of Python. Therefore, when installing PyTorch, it's advisable to use a version of Python that is one to three releases older. For instance, if the latest version of Python is 3.13, using Python 3.10, 3.11, or 3.12 is recommended.
> Many scientific computing libraries do not immediately support the newest version of Python. Therefore, when installing PyTorch, it's advisable to use a version of Python that is one or two releases older. For instance, if the latest version of Python is 3.13, using Python 3.10 or 3.11 is recommended.
Next, activate your new virtual environment (you have to do it every time you open a new terminal window or tab):

View File

@ -70,6 +70,8 @@ uv sync --dev --python 3.11
> If you have problems with the following commands above due to certain dependencies (for example, if you are using Windows), you can always fall back to regular pip:
> `uv add pip`
> `uv run python -m pip install -U -r requirements.txt`
>
> Since the TensorFo