5 Commits

Author	SHA1	Message	Date
Kasen	7bd36dccb4	Improve BPE vocabulary saving and pair frequency handling (#539)	2025-02-19 09:51:04 -06:00
Kasen	b47884ced0	Fix incorrect indentation (#536)	2025-02-18 14:47:31 -06:00
Austin Welch	0f35e370ed	fix: preserve newline tokens in BPE encoder (#495) ... * fix: preserve newline tokens in BPE encoder * further fixes * more fixes --------- Co-authored-by: rasbt <mail@sebastianraschka.com>	2025-01-21 12:47:15 -06:00
Daniel Kleine	60acb94894	BPE: fixed typo (#492) ... * fixed typo * use rel path if exists * mod gitignore and use existing vocab files --------- Co-authored-by: rasbt <mail@sebastianraschka.com>	2025-01-20 20:49:53 -06:00
Sebastian Raschka	0d4967eda6	Implementingthe BPE Tokenizer from Scratch (#487)	2025-01-17 12:22:00 -06:00