yujunjun/LLMs-from-scratch
1
0
Fork 0
mirror of https://github.com/rasbt/LLMs-from-scratch.git synced 2025-09-25 16:17:10 +00:00
Code Issues Packages Projects Releases Wiki Activity
LLMs-from-scratch/ch02/05_bpe-from-scratch/README.md
Sebastian Raschka a22d612be6
Bonus material: extending tokenizers (#496) 
* Bonus material: extending tokenizers

* small wording update
2025-01-22 09:26:54 -06:00

201 B
Raw Permalink Blame History

Byte Pair Encoding (BPE) Tokenizer From Scratch

  • bpe-from-scratch.ipynb contains optional (bonus) code that explains and shows how the BPE tokenizer works under the hood.