yujunjun/LLMs-from-scratch
1
0
Fork 0
mirror of https://github.com/rasbt/LLMs-from-scratch.git synced 2025-10-24 06:20:19 +00:00
Code Issues Packages Projects Releases Wiki Activity
LLMs-from-scratch/ch05/09_extending-tokenizers
History
Sebastian Raschka c21bfe4a23
Add PyPI package (#576) 
* Add PyPI package

* fixes

* fixes
2025-03-23 19:28:49 -05:00
..
extend-tiktoken.ipynb
Add PyPI package (#576)
2025-03-23 19:28:49 -05:00
README.md
Bonus material: extending tokenizers (#496)
2025-01-22 09:26:54 -06:00

README.md

Extending the Tiktoken BPE Tokenizer with New Tokens

  • extend-tiktoken.ipynb contains optional (bonus) code to explain how we can add special tokens to a tokenizer implemented via tiktoken and how to update the LLM accordingly