zh-cn/the-bpe-tokenizer/ #14
giscus[bot]
bot
Replies: 1 comment 1 reply
-
|
所以如果对中文使用BPE算法的话需要先分词？还是直接全部拆成单字？ |
Beta Was this translation helpful? Give feedback.
1 reply
-
|
不用先分词，全部拆成单字再用 BPE 即可，注意空格应该看作是一个特殊的”字“（SentencePiece 就是这么做的）。也有看到拆成一个个字节，然后做 Byte-Level BPE 的 |
Beta Was this translation helpful? Give feedback.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
zh-cn/the-bpe-tokenizer/
介绍了 BPE 算法用来分词的原理
https://martinlwx.github.io/zh-cn/the-bpe-tokenizer/
Beta Was this translation helpful? Give feedback.
All reactions