Sebastian Raschka
bc6f335526
Olmo 3 from scratch ( #914 )
...
* Olmo 3 from scratch
* update
* update
* update
2025-11-22 22:42:18 -06:00
Sebastian Raschka
398b079efa
RoPE decay plot ( #910 )
...
* RoPE decay plot
* RoPE decay plot
2025-11-17 17:29:49 -06:00
Sebastian Raschka
28a8408d4d
Update README wrt multi-query attention
...
Clarified the implications of using multi-query attention on modeling performance and memory usage.
2025-11-17 16:39:32 -06:00
Sebastian Raschka
a4094470c7
Write-up on how to get the most out of this book ( #909 )
2025-11-12 20:20:48 -06:00
casinca
7d92267170
fix(GatedDeltaNet): Init param A from log of a uniform distrib ( #906 )
2025-11-09 14:22:52 -06:00
rasbt
35354fac80
Use consistent title case
2025-11-06 15:22:24 -06:00
Sebastian Raschka
58f45ae5a7
Fix empty device issue ( #904 )
2025-11-05 20:04:44 -06:00
Sebastian Raschka
bcc73f731d
n_heads × d_head -> d_head × d_head in DeltaNet ( #903 )
...
Clarified the explanation of the memory size calculation for `KV_cache_DeltaNet` and updated the quadratic term from `n_heads × d_head` to `d_head × d_head`.
2025-11-05 18:28:37 -06:00
Sebastian Raschka
488bef7e3f
Image resizing
2025-11-02 21:05:38 -06:00
Sebastian Raschka
c6b8332a59
Gated DeltaNet write-up ( #901 )
...
* Gated DeltaNet write-up
* Add copyright and source information to script
Added copyright notice and source information.
* Remove unused import of Path in plot_memory_estimates
* Fix url
2025-11-02 21:03:42 -06:00
Sebastian Raschka
d6c3990c57
Training on MPS in PyTorch 2.9 ( #900 )
...
* Training on MPS in PyTorch 2.9
* update
2025-11-01 16:55:09 -05:00
Aviral Garg
27d52d6378
Fix MHAEinsum weight dimension bug when d_in != d_out ( #857 ) ( #893 )
...
* Fix MHAEinsum weight dimension bug when d_in != d_out (#857 )
Previously MHAEinsum initialized weight matrices with shape (d_out, d_in) and used inappropriate einsum notation, causing failures for non-square input-output dimensions. This commit corrects weight initialization to shape (d_in, d_out), updates einsum notation to 'bnd,do->bno', and adds three unit tests to verify parity across different d_in and d_out settings. All tests pass successfully.
* use pytest
* Update .gitignore
---------
Co-authored-by: rasbt <mail@sebastianraschka.com>
2025-10-31 21:45:31 -05:00
Sebastian Raschka
b1db33b384
simplify uv command ( #898 )
2025-10-31 19:44:57 -05:00
Sebastian Raschka
760f4c9ecc
Add bonus dependencies to pyproject ( #897 )
...
* Add bonus dependencies to pyproject
* update
2025-10-28 20:36:21 -05:00
Sebastian Raschka
0adb5b8c65
Fix ffn link ( #892 )
...
* Fix ffn link
* Apply suggestion from @rasbt
* Apply suggestion from @rasbt
2025-10-21 21:19:44 -05:00
Sebastian Raschka
7ca7c47e4a
Make quote style consistent ( #891 )
2025-10-21 19:42:33 -05:00
casinca
9276edbc37
- docs(moe): correct arXiv link for DeepSeekMoE ( #890 )
...
- docs(moe): correct paper name for 2022
2025-10-20 19:29:06 -05:00
Sebastian Raschka
218221ab62
Mixture-of-Experts intro ( #888 )
2025-10-19 22:17:59 -05:00
Sebastian Raschka
27b6dfab9e
Make it easier to toggle between thinking and instruct variants ( #887 )
2025-10-16 20:37:31 -05:00
Sebastian Raschka
7fe4874dda
Update the compression rate comment in MLA ( #883 )
...
* compression comment
* update
2025-10-14 11:10:06 -05:00
Sebastian Raschka
b969b3ef7a
Use figure numbers in ch05-7 ( #881 )
2025-10-13 16:26:35 -05:00
Sebastian Raschka
bf039ff3dc
Add alternative attention structure ( #880 )
2025-10-13 14:31:13 -05:00
Sebastian Raschka
6eb6adfa33
sliding window attention ( #879 )
2025-10-12 22:13:20 -05:00
Sebastian Raschka
21f0617ea3
Add other appendices for completeness ( #878 )
...
* Add other appendices for completeness
* update
* update
* Update
2025-10-12 19:04:53 -05:00
rasbt
44eda5340a
rm plot
2025-10-12 08:55:03 -05:00
Sebastian Raschka
9b9586688d
Multi-Head Latent Attention ( #876 )
...
* Multi-Head Latent Attention
* update
2025-10-11 20:08:30 -05:00
Sebastian Raschka
bf27ad1485
Use GB instead of GiB consistently ( #875 )
2025-10-11 09:11:33 -05:00
Sebastian Raschka
c814814d72
Grouped-Query Attention memory ( #874 )
...
* GQA memory
* remove redundant code
* update links
* update
2025-10-11 08:44:19 -05:00
rasbt
b8e12e1dd1
Use inference_device
2025-10-09 10:59:17 -05:00
Sebastian Raschka
fecfdd16ff
Add simpler BPE, and make previous BPE better ( #870 )
...
* Add simpler BPE, and make previous BPE better
* update
* Update README.md
2025-10-08 22:22:34 -05:00
Sebastian Raschka
1164cb3e8f
Qwen3 and evaluation bonus materials ( #869 )
2025-10-08 18:22:19 -05:00
Sebastian Raschka
7bd263144e
Switch from urllib to requests to improve reliability ( #867 )
...
* Switch from urllib to requests to improve reliability
* Keep ruff linter-specific
* update
* update
* update
2025-10-07 15:22:59 -05:00
Sebastian Raschka
8552565bda
Add missing comma in imports in README ( #865 )
2025-10-06 16:03:04 -05:00
Sebastian Raschka
7084123d10
Note about output dimensions ( #862 )
2025-10-01 10:47:04 -05:00
Sebastian Raschka
4d9f9dcb6c
Update ollama address ( #861 )
2025-09-30 21:05:53 -05:00
casinca
00c240ff87
some typo fixes ( #858 )
...
* fix(typo): correct scaling
* fix(typo): correct comment for `instruct`
2025-09-30 11:18:02 -05:00
Sebastian Raschka
458f2d9b67
Test dependencies with Python 3.13 ( #843 )
...
* Custom python 3.13 entry in pyproject.toml
* amend
* update
* update
* update
* Update pyproject.toml
* Update pyproject.toml
* Update pyproject.toml
* update
2025-09-27 08:38:07 -05:00
Sebastian Raschka
47867bc1cb
Update generate script ( #847 )
...
* Custom python 3.13 entry in pyproject.toml
* amend
* Update generate script
* update
* Update pyproject.toml
2025-09-27 08:03:54 -05:00
Sebastian Raschka
9bc827ea7e
Numerically stable generate on mps ( #849 )
...
* Numerically stable generate on mps
* add file
2025-09-26 22:42:44 -05:00
Sebastian Raschka
f492c949d3
Requirements update ( #851 )
...
* Requirements update
* Code change to tricker workers
* update
2025-09-26 22:19:57 -05:00
Sebastian Raschka
322000d833
Windows compile ( #845 )
...
* Custom python 3.13 entry in pyproject.toml
* amend
* Note about compile on Windows
* update
2025-09-26 12:01:19 -05:00
Sebastian Raschka
3b83705988
Update package dependencies ( #842 )
2025-09-22 18:32:39 -05:00
Sebastian Raschka
e742d8af2c
Improve MoE implementation ( #841 )
2025-09-22 15:21:06 -05:00
Sebastian Raschka
20041fb94b
Note about devcontainer root usage ( #833 )
2025-09-21 11:12:44 -05:00
Sebastian Raschka
2aa8e8130d
Note about RoPE usage ( #839 )
...
* Note about devcontainer root usage
* Add note about RoPE implementation
2025-09-20 16:25:58 +00:00
casinca
42c130623b
Qwen3Tokenizer fix for Qwen3 Base models and generation mismatch with HF (#828 )
...
* prevent `self.apply_chat_template` being applied for base Qwen models
* - added no chat template comparison in `test_chat_wrap_and_equivalence`
- removed duplicate comparison
* Revert "- added no chat template comparison in `test_chat_wrap_and_equivalence`"
This reverts commit 3a5ee8cfa19aa7e4874cd5f35171098be760b05f.
* Revert "prevent `self.apply_chat_template` being applied for base Qwen models"
This reverts commit df504397a8957886c6d6d808615545e37ceffcad.
* copied `download_file` in `utils` from https://github.com/rasbt/reasoning-from-scratch/blob/main/reasoning_from_scratch/utils.py
* added copy of test `def test_tokenizer_equivalence()` from `reasoning-from-scratch` in `test_qwen3.py`
* removed duplicate code fragment in`test_chat_wrap_and_equivalence`
* use apply_chat_template
* add toggle for instruct model
* Update tokenizer usage
---------
Co-authored-by: rasbt <mail@sebastianraschka.com>
2025-09-17 08:14:11 -05:00
Synix
bfc6389fab
fix code comment ( #834 )
2025-09-17 01:36:02 +00:00
Sebastian Raschka
b6cd0a312f
More efficient angles computation in RoPE ( #830 )
2025-09-16 03:23:33 +00:00
Sebastian Raschka
147dc49ab5
rename eval method ( #832 )
2025-09-16 02:47:20 +00:00
Sebastian Raschka
8add26cbe9
Improve weight tying handling ( #826 )
...
* Improve weight tying handling
* fix
2025-09-14 15:46:48 -05:00
