yujunjun/LLMs-from-scratch
1
0
Fork 0
mirror of https://github.com/rasbt/LLMs-from-scratch.git synced 2025-11-17 10:34:50 +00:00
Code Issues Packages Projects Releases Wiki Activity
LLMs-from-scratch/ch02/05_bpe-from-scratch
History
Sebastian Raschka fecfdd16ff
Add simpler BPE, and make previous BPE better (#870) 
* Add simpler BPE, and make previous BPE better

* update

* Update README.md
2025-10-08 22:22:34 -05:00
..
bpe-from-scratch-simple.ipynb
Add simpler BPE, and make previous BPE better (#870)
2025-10-08 22:22:34 -05:00
bpe-from-scratch.ipynb
Add simpler BPE, and make previous BPE better (#870)
2025-10-08 22:22:34 -05:00
README.md
Add simpler BPE, and make previous BPE better (#870)
2025-10-08 22:22:34 -05:00
tests.py
Add simpler BPE, and make previous BPE better (#870)
2025-10-08 22:22:34 -05:00

README.md

Byte Pair Encoding (BPE) Tokenizer From Scratch

  • bpe-from-scratch-simple.ipynb contains optional (bonus) code that explains and shows how the BPE tokenizer works under the hood; this is geared for simplicity and readability.

  • bpe-from-scratch.ipynb implements a more sophisticated (and much more complicated) BPE tokenizer that behaves similarly as tiktoken with respect to all the edge cases; it also has additional funcitionality for loading the official GPT-2 vocab.