chore: setup tokenization

[HonkingGoose]

Changes:

• Copy/paste @TWiStErRob's regex

Context:

I first started by copy/pasting the example regex from the Material for MkDocs manual, and then @TWiStErRob had a better regex for us to try.

Read the Material for MkDocs about separator.

Helps with issue #264.

[HonkingGoose]

I've tested the search in my Codespaces preview. Search work fine as far as I can tell. I have not seen anything that's obviously broken.

I'll let the maintainers review and eventually merge this. 😉

[rarkins]

How can we verify it's working as intended?

[TWiStErRob]

@rarkins expand the thread above, there are screenshots from before merge (there may be some in the issue too), if you get better results, it's working :)

Also in general just search for random things you can think of and if you can find it, it works.

[rarkins]

I happened to use the docs site for regular reasons and wonder if this update has broken some important search capabilities by tokenizing too much? For example:

[TWiStErRob]

Hmm, it looks like it needs to be able to split on something:

In your search postu is a word, but that word doesn't exist in the indexes.

[rarkins]

We might live with that if it weren't for the fact that an exact match returns no result!

[TWiStErRob]

Before this change, exact (case sensitively!) returned no result either: #265 (comment)
Right now, only "exact lowercase" match is empty.
So I think you want to disable that part only, see #264 (comment)

[rarkins]

Oh, I didn't realize that! Guess I never tried it before, or that I only needed to type part of the word before clicking

[rarkins]

I'll close my PR for now