Tokens with space have different behavior with local & remote

[bitcity]

This is related to #238 however I've noticed some inconsistency in the behavior between local & remote match. I'll reuse the example from #238.

I have spaces in my datums. For instance:
{ value:"apple",tokens:["apple"] }, { value:"great ape",tokens:["great ape"] }

When typing "great" the suggestion returns "great ape". But when typing "great " (with a space), no suggestions are returned.

The main issue here is that I WANT the token to HAVE a space. I don't want this behavior of typing a search like "ap" and getting "great ape" in the results.

If I rapidly type "great " (great with tailing space), the behavior is as described above (no suggestions are shown). However if I type "great" and wait for the remove suggestions to load, and then type a space, the suggestions are shown i.e. once they are cached locally, they will persist.

Without waiting for fix in 0.11 or 1.0, what's the place to patch this?

[jharding]

I'm a little confused because remote doesn't deal with tokenization at all. It just renders what's returned by the server.

Now if you wanted to support spaces in your tokens, you could do that. You'd just have to write a custom tokenizer.

[3m0]

If I change my tokenizer to

datumTokenizer: function (datum)
{
    return datum.value;
}

It should work, right? However, if I use this same example and search for "great a", it doesn't work.