mirror of
https://github.com/rasbt/LLMs-from-scratch.git
synced 2025-08-12 18:51:57 +00:00
Extending the Tiktoken BPE Tokenizer with New Tokens
- extend-tiktoken.ipynb contains optional (bonus) code to explain how we can add special tokens to a tokenizer implemented via
tiktokenand how to update the LLM accordingly