fix: better tokenization

[rarkins]

fix search

[TWiStErRob]

Wait, don't revert the whole thing, you just want to remove the case-switch tokenization.

[TWiStErRob]

-separator: '[\s\-,:!?=\[\]()<>{}"/\\]+|(?!\b)(?=[A-Z][a-z])|\.(?!\d)|&[lg]t;'
+separator: '[\s\-,:!?=\[\]()<>{}"/\\]+|\.(?!\d)|&[lg]t;'

[TWiStErRob]

The part (?!\b)(?=[A-Z][a-z]) is responsible for splitting up fooBar into foo and bar, which gives relevant results when people don't know the whole thing. e.g.

this was not possible before either, so removing just this small part will restore what you were used to in addition keeping the majority of the fix for #264

[rarkins]

@HonkingGoose @viceice wdyt?

[HonkingGoose]

I'll defer to people that actually know regex, I just copy pasted the regex from @TWiStErRob 😄

If you know the exact term, you should be able to find it with the search. And you should also be able to find partial matches.