Added tokenizers-remote

[jafioti]

Using a remote tokenizer from huggingface isn't advisable in prod, and so the features to enable that aren't necessarily needed all the time. This PR introduces a feature flag tokenizers-remote which is on by default. When turned off, the Remote option for tokenizers is disabled.

The main win here for me at least is the removal of OpenSSL in a production environment. I deploy to various environments, many of which have no OpenSSL installation, and since I don't use a remote tokenizer, it's unnecessary.

[philpax]

Fantastic, thank you! I actually encountered this as well the other day, but was trying to fix it by making the tokenizers dependency entirely optional - in hindsight, I should have checked for an option on tokenizers itself 😅

I'll test this and merge it in if all's well.

[jafioti]

I just need to fix one issue, I think the model crates are pulling in llm-base with it's default features, so I don't think there's a way to turn off the tokenizers-remote feature yet from the outside.

[jafioti]

Ok, fixed. I removed the feature as default from llm-base, but it's still turned on by default from llm, so the model crates don't need to change.

Now from the outside if you pull in the crate like so: llm = {version="1", default-features=false, features=["llama"]} no OpenSSL is included.

[philpax]

Thanks once again! Much appreciated, this will be handy for deployment on OpenSSL-less environments :)