mirror of
https://github.com/rasbt/LLMs-from-scratch.git
synced 2025-06-26 23:50:03 +00:00
f63f04d8d5
* Fix BPE bonus materials * fix bpe implementation * update * Add 'Hello, world. Is this-- a test?' test case * update link to test file * update path handling * update path handling * fix pytest paths
Chapter 2: Working with Text Data
- compare-bpe-tiktoken.ipynb benchmarks various byte pair encoding implementations
- bpe_openai_gpt2.py is the original bytepair encoder code used by OpenAI