3 Commits

Author	SHA1	Message	Date
Austin Welch	0f35e370ed	fix: preserve newline tokens in BPE encoder (#495) ... * fix: preserve newline tokens in BPE encoder * further fixes * more fixes --------- Co-authored-by: rasbt <mail@sebastianraschka.com>	2025-01-21 12:47:15 -06:00
Daniel Kleine	60acb94894	BPE: fixed typo (#492) ... * fixed typo * use rel path if exists * mod gitignore and use existing vocab files --------- Co-authored-by: rasbt <mail@sebastianraschka.com>	2025-01-20 20:49:53 -06:00
Sebastian Raschka	0d4967eda6	Implementingthe BPE Tokenizer from Scratch (#487)	2025-01-17 12:22:00 -06:00