mirror of
https://github.com/rasbt/LLMs-from-scratch.git
synced 2025-08-21 07:02:49 +00:00
3 lines
256 B
Markdown
3 lines
256 B
Markdown
# Extending the Tiktoken BPE Tokenizer with New Tokens
|
|
|
|
- [extend-tiktoken.ipynb](extend-tiktoken.ipynb) contains optional (bonus) code to explain how we can add special tokens to a tokenizer implemented via `tiktoken` and how to update the LLM accordingly |