Add preprocessor to patch PromptGuard scores for inserted characters

[cynikolai]

Problem: Inserting spaces between characters in given prompts causes misclassifications in PromptGuard. See meta-llama/llama-models#50 for more context.

Solution: Tokenize the string with all spaces removed, to ensure that larger tokens (for example, [“ignore”]) are not broken up into smaller tokens (for example, [“i”, “g”, “n”, “o”, “r”, “e”] . Add back spaces between the larger tokens if spaces exist in the original string.

This approach showed a slight positive impact on all of our evaluation datasets, suggesting that making the system more robust to jailbreaks that disrupt tokenization like this one will be an important part of improving model quality. Notably, simply subtracting spaces from the string lead to a moderate quality regression on some datasets, which is why we don’t take that simpler approach here.

This solution only targets jailbreaks enabled by inserted spaces and not other special characters. For a more complete approach longer term, we’re continuing to work on building more adversarial examples into our dataset.

The preprocessor is used by default by our inference utilities.

[mreso]

LGTM but this might be easily circumvented if the tokenizer is not lossless (see comment)

[mreso]

IIRC tokenizer are not always lossless. Could this be circumvented by adding a word/string combination into the text which can not be tokenized "losslessly" leading to a ValueError being raised in .index() as the "detokenized" string can not be found?

[mreso]

Turns out tiktoken is actually lossless.