yujunjun/LLMs-from-scratch
1
0
Fork 0
mirror of https://github.com/rasbt/LLMs-from-scratch.git synced 2025-08-28 18:40:01 +00:00
Code Issues Packages Projects Releases Wiki Activity
LLMs-from-scratch/ch02/05_bpe-from-scratch
History
Sebastian Raschka 02ca4ac42d BPE cosmetics (#629) 
* Llama3 from scratch improvements

* Cosmetic BPE improvements

* restore

* Update ch02/05_bpe-from-scratch/bpe-from-scratch.ipynb

* Update ch02/05_bpe-from-scratch/bpe-from-scratch.ipynb

* endoftext whitespace
2025-04-18 18:57:09 -05:00
..
tests
Fix BPE bonus materials (#561)
2025-03-08 17:21:30 -06:00
bpe-from-scratch.ipynb
BPE cosmetics (#629)
2025-04-18 18:57:09 -05:00
README.md
Bonus material: extending tokenizers (#496)
2025-01-22 09:26:54 -06:00

README.md

Byte Pair Encoding (BPE) Tokenizer From Scratch

  • bpe-from-scratch.ipynb contains optional (bonus) code that explains and shows how the BPE tokenizer works under the hood.