mirror of
https://github.com/rasbt/LLMs-from-scratch.git
synced 2025-09-02 12:57:41 +00:00
8 lines
249 B
Markdown
8 lines
249 B
Markdown
|
# Chapter 2: Working with Text Data
|
||
|
|
|
||
|
|
|
||
|
|
|
||
|
|
- [compare-bpe-tiktoken.ipynb](compare-bpe-tiktoken.ipynb) benchmarks various byte pair encoding implementations
|
||
|
|
- [bpe_openai_gpt2.py](bpe_openai_gpt2.py) is the original bytepair encoder code used by OpenAI
|
||
|
|
|