yujunjun/LLMs-from-scratch
1
0
Fork 0
mirror of https://github.com/rasbt/LLMs-from-scratch.git synced 2025-11-01 02:10:15 +00:00
Code Issues Packages Projects Releases Wiki Activity
LLMs-from-scratch/ch05/09_extending-tokenizers
History
Sebastian Raschka dcaac28b92 Bonus material: extending tokenizers (#496) 
* Bonus material: extending tokenizers

* small wording update
2025-01-22 09:26:54 -06:00
..
extend-tiktoken.ipynb
Bonus material: extending tokenizers (#496)
2025-01-22 09:26:54 -06:00
gpt_download.py
Bonus material: extending tokenizers (#496)
2025-01-22 09:26:54 -06:00
previous_chapters.py
Bonus material: extending tokenizers (#496)
2025-01-22 09:26:54 -06:00
README.md
Bonus material: extending tokenizers (#496)
2025-01-22 09:26:54 -06:00

README.md

Extending the Tiktoken BPE Tokenizer with New Tokens

  • extend-tiktoken.ipynb contains optional (bonus) code to explain how we can add special tokens to a tokenizer implemented via tiktoken and how to update the LLM accordingly