fix: preserve newline tokens in BPE encoder (#495) · rasbt/LLMs-from-scratch@0f35e37 · GitHub

Commit

fix: preserve newline tokens in BPE encoder (#495)

Browse files

* fix: preserve newline tokens in BPE encoder

* further fixes

* more fixes

---------

Co-authored-by: rasbt <[email protected]>

Loading branch information

austinmw and rasbt authored Jan 21, 2025

1 parent 60acb94 commit 0f35e37