feat: add new textspliter option: lenfunc

[whyiug]

I want to add a new parameter to the textspliter struct,lenfunc, to represent a custom length function. In this way, sentences can be splited according to lenfuc, for example, utf8.RuneCountInString can be passed to split Chinese characters.
Also aligned with the python code https://github.com/langchain-ai/langchain/blob/00a09e1b7117f3bde14a44748510fcccc95f9de5/libs/langchain/langchain/text_splitter.py.
For me, it's a must-have feature.

[whyiug]

Also satisfy issue #231

[whyiug]

ping @tmc

[tmc]

Hmm, it might be best to just default to RuneCount/RuneCountInString to treat content more naturally, thoughts?

I think I prefer this over erroneously using len() on strings.

[whyiug]

Your suggestion is feasible, just like what is done in Python projects.
// In Python, the len function indeed represents the number of characters rather than the number of bytes.
I have already updated the latest code. Please review it.

[tmc]

This is better, but I can't help but think we shouldn't even make this configurable and just use utf8.RuneCountInString -- thoughts?

[tmc]

this could just be utf8.RuneCountInString instead of being wrapped with defaultLenFunc, I believe.

[whyiug]

Hmm, I don't think so.
Some scenarios require custom length functions.
Let's say I want to have the same number of words, not characters in each chunk.
I can simply set the LenFunc:

func countWords(s string) int {
    words := strings.Fields(s)
    return len(words)
}

Or the number of tokens divided by other word dividers.

[tmc]

LGTM