Ignore tokens if their IDs in added token list are below the vocab's base size#3585

Closed

seungduk-yanolja wants to merge 5 commits intoggml-org:masterfrom seungduk-yanolja:master

+21-18

Commits on Oct 11, 2023

Commits on Oct 12, 2023

simply ignore added tokens that id < vocab size
seungduk-yanolja
committed

Commits on Oct 15, 2023